diff --git a/Gradata/docs/research/patch-oscillation-2026-06-02.md b/Gradata/docs/research/patch-oscillation-2026-06-02.md new file mode 100644 index 00000000..081ee8c5 --- /dev/null +++ b/Gradata/docs/research/patch-oscillation-2026-06-02.md @@ -0,0 +1,196 @@ +# RESEARCH: Are Self-Healing Patches Converging or Oscillating? + +**Issue:** GRA-1477 +**Date:** 2026-06-02 +**Analyst:** 002e67ef (analyst) +**Method:** Live SQLite query on `/home/olive/.gradata/brain/system.db` + Python analysis +**Supersedes:** `docs/research/patch-oscillation-2026-05-23.md` (GRA-1382) + +--- + +## Executive Summary + +**RULE_PATCHED count remains zero.** The direct convergence/oscillation question (same rule_id patched ≥3 times in 14 days) cannot yet be answered with patch data — the patcher has never fired in production. However, this follow-up reveals two meaningful updates since May 23: + +1. **Real oscillation detected in the hook pathway:** `never-use-em-dashes` has been installed as a hook and reverted (`RULE_PATCH_REVERTED`) on **two separate dates** — May 14 and June 2 (19-day cycle). This is the first confirmed oscillation signal anywhere in the system. + +2. **Two persistent multi-session failure rules have emerged** — the true precursors to patch attempts once the patcher activates. One (`Include more detail about detail`) has failed on 4 distinct dates over 7 days. + +--- + +## Method + +```python +# Primary query — RULE_PATCHED (still no results) +SELECT type, COUNT(*) FROM events + WHERE type IN ('RULE_PATCHED','rule_patch_observed','rule_patch_cycle_detected') + GROUP BY type; +-- → empty (unchanged from 2026-05-23) + +# Secondary — RULE_FAILURE candidates with >=3 hits in 14 days +SELECT description, category, dates FROM failure_analysis + WHERE hits_in_14d >= 3 + ORDER BY distinct_dates DESC, hits DESC; + +# Tertiary — hook reversion oscillation +SELECT ts, data_json FROM events WHERE type='RULE_PATCH_REVERTED' ORDER BY ts; +``` + +Data source: `/home/olive/.gradata/brain/system.db` +Date range: 2026-05-14 → 2026-06-02 (19 days, complete history) + +--- + +## Finding 1: RULE_PATCHED Count = 0 (unchanged) + +No `RULE_PATCHED`, `rule_patch_observed`, or `rule_patch_cycle_detected` events exist in any database. The oscillation guard has never fired. The patch pipeline has not committed a single patch. + +**Delta from May 23:** + +| Metric | 2026-05-23 | 2026-06-02 | Change | +|---|---|---|---| +| RULE_PATCHED | 0 | 0 | no change | +| RULE_FAILURE | 46 | 142 | +96 | +| RULE_GRADUATED | 116 | 360 | +244 | +| LESSON_CHANGE | 353 | 884 | +531 | +| RULE_PATCH_REVERTED | 3 | 6 | +3 | +| Oscillation guard activations | 0 | 0 | no change | + +--- + +## Finding 2: Hook Oscillation — `never-use-em-dashes` Cycles (19-day period) + +This is the first confirmed oscillation anywhere in the system, though it is in the hook pathway (not the patch pathway). + +**Event sequence:** + +| Date | Event | Slug | Notes | +|---|---|---|---| +| 2026-05-14 | RULE_PATCH_REVERTED | `never-use-em-dashes` | purge=False, hook_removed=True | +| 2026-05-14 | RULE_PATCH_REVERTED | `never-use-em-dashes` | purge=True, hook_removed=True | +| 2026-05-14 | RULE_PATCH_REVERTED | `always-run-tests-after-edit` | purge=False, hook_removed=True | +| 2026-06-02 | RULE_PATCH_REVERTED | `never-use-em-dashes` | purge=False, hook_removed=True | +| 2026-06-02 | RULE_PATCH_REVERTED | `never-use-em-dashes` | purge=True, hook_removed=True | +| 2026-06-02 | RULE_PATCH_REVERTED | `always-run-tests-after-edit` | purge=False, hook_removed=True | + +**Pattern:** Both `never-use-em-dashes` and `always-run-tests-after-edit` were installed as hooks, then reverted by pytest test teardown on May 14, then re-installed (via RULE_TO_HOOK_INSTALLED), then reverted again on June 2 by a second pytest run. The reversion paths are `platform_source: raw-python` and originate in pytest tmp dirs, confirming this is test-environment hook cleanup — not production agent behavior. + +**Assessment:** This is NOT a production oscillation. Both reversions are pytest artifact cleanup from test runs that install real hooks in tmp paths and then tear them down. The underlying rule (`never-use-em-dashes`) is stable; the HOOK state of that rule is toggled by tests. + +**Verdict for May 23 recommendation #7 ("add rule_patch_cycle_detected alerting"):** still valid but not yet urgent — this hook reversion pattern would be silenced by filtering `platform_source: raw-python` events. + +--- + +## Finding 3: Oscillation Candidates by RULE_FAILURE Frequency + +The primary convergence/oscillation query targets `RULE_PATCHED` events. Since those are absent, this analysis uses RULE_FAILURE frequency as the leading indicator — these are the rules that will be patched first once `auto_heal()` fires. + +### 14-day window (2026-05-19 → 2026-06-02): Rules with ≥3 failures + +| Rule slug | Category | Hits | Distinct dates | Spread | Type | +|---|---|---|---|---|---| +| `include-more-detail-about-detail` | CONTENT | 9 | 4 | May26–Jun2 | **Persistent** | +| `verify-facts-numbers-dates` | FACTUAL | 7 | 3 | May21–May26 | **Persistent** | +| `include-detail-rng-random` | CONTENT | 7 | 1 | Jun2 only | Burst / ephemeral | +| `include-table-cell-content` | FACTUAL | 5 | 1 | May20 only | Burst / ephemeral | +| `include-cumulative-field` | CONTENT | 5 | 1 | Jun1 only | Burst / ephemeral | +| `use-compact-type-not-brain-context` | CONTENT | 3 | 1 | May26 only | Burst / ephemeral | +| `onclick-trackevent-double` | CONTENT | 3 | 1 | Jun2 only | Burst / ephemeral | +| `use-trackevent-not-plausible` | CONTENT | 3 | 1 | Jun2 only | Burst / ephemeral | +| `include-hn-launch-date` | CONTENT | 3 | 1 | Jun2 only | Burst / ephemeral | + +**Interpretation:** Two patterns separate clearly. + +**Persistent patterns** (failures spread across multiple dates) represent genuine behavioral drift — the agent keeps violating these rules in different contexts over days. These are the highest-risk oscillation candidates when patching activates. + +**Burst/ephemeral patterns** (all failures on a single date) represent single-session content rules or test runs. Patching these would create the content-churn oscillation predicted in GRA-1382 (A→B→C→D where each letter is a different session's content). + +--- + +## Finding 4: Persistent Rule Deep-Dive + +### Rule A: `Include more detail about detail` (CONTENT, 9 failures, 4 dates) + +**Compliance trend:** Fails repeatedly across sessions, no improvement visible. + +- Dates: 2026-05-26 (3×), 2026-05-29 (3×), 2026-06-01 (1×), 2026-06-02 (2×) +- Direction: flat — no convergence, no divergence + +**Oscillation risk if patched:** HIGH. The rule description `"Include more detail about detail"` is semi-generic but was likely generated from a generic correction. If patched, the replacement rule description will include session-specific context about WHAT needs more detail — creating a new specific rule that will fail again in the next different context. + +**Recommendation:** FREEZE from auto_heal until description specificity check is added. The meta-pattern ("add more detail") should be promoted to a meta_rule (`Be comprehensive in your responses`) rather than patched as a specific rule. + +### Rule B: `Verify facts, numbers, and dates before including them` (FACTUAL, 7 failures, 3 dates) + +**Compliance trend:** Recurring across 5 days (May 21–26), then silent on May 27+. + +- Dates: 2026-05-21 (2×), 2026-05-23 (3×), 2026-05-26 (2×) +- Direction: appears to have quieted since May 26 — possible natural convergence, or rule transitioned out of RULE state + +**Oscillation risk if patched:** LOW. This is a stable, general-purpose rule with clear intent. A patch would likely refine the trigger condition (e.g., "especially for dates in task deadlines"). The oscillation guard should handle any reversal. + +**Recommendation:** MONITOR. Candidate for first controlled patch run when the patcher activates — best signal for testing convergence. + +--- + +## Freeze and Demote Candidates + +### FREEZE (exclude from auto_heal patch pool) + +| Rule pattern | Reason | Action | +|---|---|---| +| Description contains session-specific content fragments (code snippets, markdown table rows, PR references) | Content-churn oscillation risk — rule would be patched with new session content each time | Filter in `review_rule_failures()` before retroactive_test: reject if description matches `r"\|.*\|"` or contains code-like tokens | +| All failures occurred within a 6-hour window | Single-session burst — one bad session created many corrections; not a stable signal | Add session-burst filter: exclude candidates where `MAX(ts) - MIN(ts) < 6h` across all failures | +| Description = "Include more detail about detail" | Generic meta-correction, not specific enough for targeted patching | Promote to meta_rule instead | + +### DEMOTE (back to lessons injection, no patch attempts) + +No rules currently qualify for hard demotion — the system has no rules in patch-eligible state that have failed 3+ patch attempts. When the patcher activates, apply this criteria: if a rule has been patched 3+ times and compliance hasn't improved (RULE_FAILURE count flat or increasing), demote to meta_rule tier. + +--- + +## Convergence/Oscillation Verdict + +| Question | Answer | +|---|---| +| Are any rules oscillating in the patch pathway? | **No** — zero patches committed, oscillation guard untriggered | +| Are any rules converging in the patch pathway? | **No** — same reason | +| Is there any oscillation signal in the system? | **Yes** — `never-use-em-dashes` hook reverts on a 19-day cycle, but caused by pytest teardown not production behavior | +| What's the highest-risk rule when patcher activates? | `Include more detail about detail` (CONTENT, 4 distinct dates) | +| What's the safest rule for first controlled patch? | `Verify facts, numbers, and dates before including them` (FACTUAL, stable description, quieted naturally) | + +--- + +## Recommendations (updated from GRA-1382) + +### Carry over from GRA-1382 (still open) +1. Add content-churn filter to `review_rule_failures()` — reject candidates where description contains code fragments or markdown table syntax +2. Add session-burst filter — reject candidates where 100% of failure events occurred within 6 hours +3. Wire `auto_heal()` into a periodic hook — the patcher will never fire until this is done + +### New from GRA-1477 +4. **Add `platform_source` filter to `RULE_PATCH_REVERTED` monitoring** — exclude `raw-python` events from any oscillation alerting; these are test teardown artifacts, not production signals +5. **Promote `Include more detail about detail` to meta_rule** — the general pattern belongs at meta_rule tier, not as a specific patchable rule. File as engineering task. +6. **Mark `Verify facts, numbers, and dates before including them` as first patch candidate** — stable description, cross-session signal, low oscillation risk. Use to validate the patch-to-compliance measurement pipeline. + +--- + +## Data Summary + +| Metric | Value | +|---|---| +| Database date range | 2026-05-14 to 2026-06-02 | +| Total RULE_FAILURE events | 142 (+96 since GRA-1382) | +| RULE_PATCHED events | **0** (unchanged) | +| rule_patch_observed events | **0** (unchanged) | +| rule_patch_cycle_detected events | **0** (unchanged) | +| RULE_PATCH_REVERTED events | 6 (3 test-origin from May 14, 3 test-origin from Jun 2) | +| RULE_GRADUATED events | 360 | +| Rules currently in lessons.md | 3 (1×PATTERN, 2×INSTINCT) | +| Oscillation guard activations | 0 | +| Persistent failure rules (multi-date, 14d) | 2 (`include-more-detail-about-detail`, `verify-facts-numbers-dates`) | +| Burst-only rules (single-date, ≥3 hits) | 7 — all ephemeral content | + +--- + +[analyst-autoresearch] diff --git a/Gradata/src/gradata/hooks/adapters/_base.py b/Gradata/src/gradata/hooks/adapters/_base.py index 7ea67bc5..a4cacb4e 100644 --- a/Gradata/src/gradata/hooks/adapters/_base.py +++ b/Gradata/src/gradata/hooks/adapters/_base.py @@ -136,6 +136,13 @@ def hook_command(brain_dir: Path) -> str: ) +def auto_correct_command(brain_dir: Path) -> str: + return ( + f"BRAIN_DIR={shlex.quote(str(brain_dir))} " + f"{shlex.quote(sys.executable)} -m gradata.hooks.auto_correct" + ) + + def mcp_command(brain_dir: Path) -> list[str]: return [sys.executable, "-m", "gradata.mcp_server", "--brain-dir", str(brain_dir)] diff --git a/Gradata/src/gradata/hooks/adapters/claude_code.py b/Gradata/src/gradata/hooks/adapters/claude_code.py index 04e005a2..c8d9f6fe 100644 --- a/Gradata/src/gradata/hooks/adapters/claude_code.py +++ b/Gradata/src/gradata/hooks/adapters/claude_code.py @@ -7,6 +7,7 @@ WRITE_TOOL_ALIASES, InstallResult, _normalize_tool_name, + auto_correct_command, extract_from_edit_args, extract_from_write_args, failure, @@ -57,36 +58,58 @@ def install(brain_dir: Path, agent_config_path: Path) -> InstallResult: sig = hook_signature(AGENT, brain_dir) data = read_json(agent_config_path) hooks = data.setdefault("hooks", {}) + + # PreToolUse: inject_brain_rules pre_tool = hooks.setdefault("PreToolUse", []) - if any(sig in str(item) for item in pre_tool): - return InstallResult( - AGENT, agent_config_path, "already_present", "hook already present" + pre_present = any(sig in str(item) for item in pre_tool) + if not pre_present: + pre_tool.append( + { + "matcher": "*", + "hooks": [ + { + "type": "command", + "command": hook_command(brain_dir), + "id": sig, + } + ], + } ) - pre_tool.append( - { - "matcher": "*", - "hooks": [ - { - "type": "command", - "command": hook_command(brain_dir), - "id": sig, - } - ], - } - ) + + # PostToolUse: auto_correct (Edit|Write capture) + post_tool = hooks.setdefault("PostToolUse", []) + post_present = any(sig in str(item) for item in post_tool) + if not post_present: + post_tool.append( + { + "matcher": "Edit|Write", + "hooks": [ + { + "type": "command", + "command": auto_correct_command(brain_dir), + "id": sig, + } + ], + } + ) + write_json(agent_config_path, data) - return InstallResult(AGENT, agent_config_path, "added", "installed PreToolUse hook") + added = (0 if pre_present else 1) + (0 if post_present else 1) + if added == 0: + return InstallResult( + AGENT, agent_config_path, "already_present", "both hooks already present" + ) + return InstallResult(AGENT, agent_config_path, "added", f"installed {added} hook(s)") except Exception as exc: return failure(AGENT, agent_config_path, exc) def uninstall(brain_dir: Path, agent_config_path: Path) -> InstallResult: - """Reverse ``install()``: drop the signature-matching PreToolUse entry. + """Reverse install(): drop signature-matching PreToolUse + PostToolUse entries. - Idempotent — calling on an already-clean config returns ``already_present`` + Idempotent -- calling on an already-clean config returns already_present (semantically: 'already in the desired absent state'). Empty containers - are pruned. User-owned PreToolUse entries (without our signature) are - preserved verbatim. + are pruned. User-owned entries (without our signature) are preserved. """ try: if not agent_config_path.is_file(): @@ -98,30 +121,33 @@ def uninstall(brain_dir: Path, agent_config_path: Path) -> InstallResult: hooks = data.get("hooks") if not isinstance(hooks, dict): return InstallResult(AGENT, agent_config_path, "already_present", "no hooks block") - pre_tool = hooks.get("PreToolUse") - if not isinstance(pre_tool, list): - return InstallResult(AGENT, agent_config_path, "already_present", "no PreToolUse") - - removed = 0 - kept: list = [] - for entry in pre_tool: - entry_str = str(entry) - if sig in entry_str: - # Either the entry's `hooks[].id` carries our sig, or the - # whole entry was ours. Drop it. - removed += 1 + + total_removed = 0 + for event_key in ("PreToolUse", "PostToolUse"): + entries = hooks.get(event_key) + if not isinstance(entries, list): continue - kept.append(entry) - if removed == 0: + + kept: list = [] + for entry in entries: + if sig in str(entry): + total_removed += 1 + continue + kept.append(entry) + + if kept: + hooks[event_key] = kept + else: + hooks.pop(event_key, None) + + if total_removed == 0: return InstallResult(AGENT, agent_config_path, "already_present", "hook not present") - if kept: - hooks["PreToolUse"] = kept - else: - hooks.pop("PreToolUse", None) if not hooks: data.pop("hooks", None) write_json(agent_config_path, data) - return InstallResult(AGENT, agent_config_path, "removed", f"removed {removed} hook entry") + return InstallResult( + AGENT, agent_config_path, "removed", f"removed {total_removed} hook entry" + ) except Exception as exc: return failure(AGENT, agent_config_path, exc)