Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
196 changes: 196 additions & 0 deletions Gradata/docs/research/patch-oscillation-2026-06-02.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
# RESEARCH: Are Self-Healing Patches Converging or Oscillating?

**Issue:** GRA-1477
**Date:** 2026-06-02
**Analyst:** 002e67ef (analyst)
**Method:** Live SQLite query on `/home/olive/.gradata/brain/system.db` + Python analysis
**Supersedes:** `docs/research/patch-oscillation-2026-05-23.md` (GRA-1382)

---

## Executive Summary

**RULE_PATCHED count remains zero.** The direct convergence/oscillation question (same rule_id patched ≥3 times in 14 days) cannot yet be answered with patch data — the patcher has never fired in production. However, this follow-up reveals two meaningful updates since May 23:

1. **Real oscillation detected in the hook pathway:** `never-use-em-dashes` has been installed as a hook and reverted (`RULE_PATCH_REVERTED`) on **two separate dates** — May 14 and June 2 (19-day cycle). This is the first confirmed oscillation signal anywhere in the system.

2. **Two persistent multi-session failure rules have emerged** — the true precursors to patch attempts once the patcher activates. One (`Include more detail about detail`) has failed on 4 distinct dates over 7 days.

---

## Method

```python
# Primary query — RULE_PATCHED (still no results)
SELECT type, COUNT(*) FROM events
WHERE type IN ('RULE_PATCHED','rule_patch_observed','rule_patch_cycle_detected')
GROUP BY type;
-- → empty (unchanged from 2026-05-23)

# Secondary — RULE_FAILURE candidates with >=3 hits in 14 days
SELECT description, category, dates FROM failure_analysis
WHERE hits_in_14d >= 3
ORDER BY distinct_dates DESC, hits DESC;

# Tertiary — hook reversion oscillation
SELECT ts, data_json FROM events WHERE type='RULE_PATCH_REVERTED' ORDER BY ts;
```

Data source: `/home/olive/.gradata/brain/system.db`
Date range: 2026-05-14 → 2026-06-02 (19 days, complete history)

---

## Finding 1: RULE_PATCHED Count = 0 (unchanged)

No `RULE_PATCHED`, `rule_patch_observed`, or `rule_patch_cycle_detected` events exist in any database. The oscillation guard has never fired. The patch pipeline has not committed a single patch.

**Delta from May 23:**

| Metric | 2026-05-23 | 2026-06-02 | Change |
|---|---|---|---|
| RULE_PATCHED | 0 | 0 | no change |
| RULE_FAILURE | 46 | 142 | +96 |
| RULE_GRADUATED | 116 | 360 | +244 |
| LESSON_CHANGE | 353 | 884 | +531 |
| RULE_PATCH_REVERTED | 3 | 6 | +3 |
| Oscillation guard activations | 0 | 0 | no change |

---

## Finding 2: Hook Oscillation — `never-use-em-dashes` Cycles (19-day period)

This is the first confirmed oscillation anywhere in the system, though it is in the hook pathway (not the patch pathway).

**Event sequence:**

| Date | Event | Slug | Notes |
|---|---|---|---|
| 2026-05-14 | RULE_PATCH_REVERTED | `never-use-em-dashes` | purge=False, hook_removed=True |
| 2026-05-14 | RULE_PATCH_REVERTED | `never-use-em-dashes` | purge=True, hook_removed=True |
| 2026-05-14 | RULE_PATCH_REVERTED | `always-run-tests-after-edit` | purge=False, hook_removed=True |
| 2026-06-02 | RULE_PATCH_REVERTED | `never-use-em-dashes` | purge=False, hook_removed=True |
| 2026-06-02 | RULE_PATCH_REVERTED | `never-use-em-dashes` | purge=True, hook_removed=True |
| 2026-06-02 | RULE_PATCH_REVERTED | `always-run-tests-after-edit` | purge=False, hook_removed=True |

**Pattern:** Both `never-use-em-dashes` and `always-run-tests-after-edit` were installed as hooks, then reverted by pytest test teardown on May 14, then re-installed (via RULE_TO_HOOK_INSTALLED), then reverted again on June 2 by a second pytest run. The reversion paths are `platform_source: raw-python` and originate in pytest tmp dirs, confirming this is test-environment hook cleanup — not production agent behavior.

**Assessment:** This is NOT a production oscillation. Both reversions are pytest artifact cleanup from test runs that install real hooks in tmp paths and then tear them down. The underlying rule (`never-use-em-dashes`) is stable; the HOOK state of that rule is toggled by tests.

**Verdict for May 23 recommendation #7 ("add rule_patch_cycle_detected alerting"):** still valid but not yet urgent — this hook reversion pattern would be silenced by filtering `platform_source: raw-python` events.

---

## Finding 3: Oscillation Candidates by RULE_FAILURE Frequency

The primary convergence/oscillation query targets `RULE_PATCHED` events. Since those are absent, this analysis uses RULE_FAILURE frequency as the leading indicator — these are the rules that will be patched first once `auto_heal()` fires.

### 14-day window (2026-05-19 → 2026-06-02): Rules with ≥3 failures

| Rule slug | Category | Hits | Distinct dates | Spread | Type |
|---|---|---|---|---|---|
| `include-more-detail-about-detail` | CONTENT | 9 | 4 | May26–Jun2 | **Persistent** |
| `verify-facts-numbers-dates` | FACTUAL | 7 | 3 | May21–May26 | **Persistent** |
| `include-detail-rng-random` | CONTENT | 7 | 1 | Jun2 only | Burst / ephemeral |
| `include-table-cell-content` | FACTUAL | 5 | 1 | May20 only | Burst / ephemeral |
| `include-cumulative-field` | CONTENT | 5 | 1 | Jun1 only | Burst / ephemeral |
| `use-compact-type-not-brain-context` | CONTENT | 3 | 1 | May26 only | Burst / ephemeral |
| `onclick-trackevent-double` | CONTENT | 3 | 1 | Jun2 only | Burst / ephemeral |
| `use-trackevent-not-plausible` | CONTENT | 3 | 1 | Jun2 only | Burst / ephemeral |
| `include-hn-launch-date` | CONTENT | 3 | 1 | Jun2 only | Burst / ephemeral |

**Interpretation:** Two patterns separate clearly.

**Persistent patterns** (failures spread across multiple dates) represent genuine behavioral drift — the agent keeps violating these rules in different contexts over days. These are the highest-risk oscillation candidates when patching activates.

**Burst/ephemeral patterns** (all failures on a single date) represent single-session content rules or test runs. Patching these would create the content-churn oscillation predicted in GRA-1382 (A→B→C→D where each letter is a different session's content).

---

## Finding 4: Persistent Rule Deep-Dive

### Rule A: `Include more detail about detail` (CONTENT, 9 failures, 4 dates)

**Compliance trend:** Fails repeatedly across sessions, no improvement visible.

- Dates: 2026-05-26 (3×), 2026-05-29 (3×), 2026-06-01 (1×), 2026-06-02 (2×)
- Direction: flat — no convergence, no divergence

**Oscillation risk if patched:** HIGH. The rule description `"Include more detail about detail"` is semi-generic but was likely generated from a generic correction. If patched, the replacement rule description will include session-specific context about WHAT needs more detail — creating a new specific rule that will fail again in the next different context.

**Recommendation:** FREEZE from auto_heal until description specificity check is added. The meta-pattern ("add more detail") should be promoted to a meta_rule (`Be comprehensive in your responses`) rather than patched as a specific rule.

### Rule B: `Verify facts, numbers, and dates before including them` (FACTUAL, 7 failures, 3 dates)

**Compliance trend:** Recurring across 5 days (May 21–26), then silent on May 27+.

- Dates: 2026-05-21 (2×), 2026-05-23 (3×), 2026-05-26 (2×)
- Direction: appears to have quieted since May 26 — possible natural convergence, or rule transitioned out of RULE state

**Oscillation risk if patched:** LOW. This is a stable, general-purpose rule with clear intent. A patch would likely refine the trigger condition (e.g., "especially for dates in task deadlines"). The oscillation guard should handle any reversal.

**Recommendation:** MONITOR. Candidate for first controlled patch run when the patcher activates — best signal for testing convergence.

---

## Freeze and Demote Candidates

### FREEZE (exclude from auto_heal patch pool)

| Rule pattern | Reason | Action |
|---|---|---|
| Description contains session-specific content fragments (code snippets, markdown table rows, PR references) | Content-churn oscillation risk — rule would be patched with new session content each time | Filter in `review_rule_failures()` before retroactive_test: reject if description matches `r"\|.*\|"` or contains code-like tokens |
| All failures occurred within a 6-hour window | Single-session burst — one bad session created many corrections; not a stable signal | Add session-burst filter: exclude candidates where `MAX(ts) - MIN(ts) < 6h` across all failures |
| Description = "Include more detail about detail" | Generic meta-correction, not specific enough for targeted patching | Promote to meta_rule instead |

### DEMOTE (back to lessons injection, no patch attempts)

No rules currently qualify for hard demotion — the system has no rules in patch-eligible state that have failed 3+ patch attempts. When the patcher activates, apply this criteria: if a rule has been patched 3+ times and compliance hasn't improved (RULE_FAILURE count flat or increasing), demote to meta_rule tier.

---

## Convergence/Oscillation Verdict

| Question | Answer |
|---|---|
| Are any rules oscillating in the patch pathway? | **No** — zero patches committed, oscillation guard untriggered |
| Are any rules converging in the patch pathway? | **No** — same reason |
| Is there any oscillation signal in the system? | **Yes** — `never-use-em-dashes` hook reverts on a 19-day cycle, but caused by pytest teardown not production behavior |
| What's the highest-risk rule when patcher activates? | `Include more detail about detail` (CONTENT, 4 distinct dates) |
| What's the safest rule for first controlled patch? | `Verify facts, numbers, and dates before including them` (FACTUAL, stable description, quieted naturally) |

---

## Recommendations (updated from GRA-1382)

### Carry over from GRA-1382 (still open)
1. Add content-churn filter to `review_rule_failures()` — reject candidates where description contains code fragments or markdown table syntax
2. Add session-burst filter — reject candidates where 100% of failure events occurred within 6 hours
3. Wire `auto_heal()` into a periodic hook — the patcher will never fire until this is done

### New from GRA-1477
4. **Add `platform_source` filter to `RULE_PATCH_REVERTED` monitoring** — exclude `raw-python` events from any oscillation alerting; these are test teardown artifacts, not production signals
5. **Promote `Include more detail about detail` to meta_rule** — the general pattern belongs at meta_rule tier, not as a specific patchable rule. File as engineering task.
6. **Mark `Verify facts, numbers, and dates before including them` as first patch candidate** — stable description, cross-session signal, low oscillation risk. Use to validate the patch-to-compliance measurement pipeline.

---

## Data Summary

| Metric | Value |
|---|---|
| Database date range | 2026-05-14 to 2026-06-02 |
| Total RULE_FAILURE events | 142 (+96 since GRA-1382) |
| RULE_PATCHED events | **0** (unchanged) |
| rule_patch_observed events | **0** (unchanged) |
| rule_patch_cycle_detected events | **0** (unchanged) |
| RULE_PATCH_REVERTED events | 6 (3 test-origin from May 14, 3 test-origin from Jun 2) |
| RULE_GRADUATED events | 360 |
| Rules currently in lessons.md | 3 (1×PATTERN, 2×INSTINCT) |
| Oscillation guard activations | 0 |
| Persistent failure rules (multi-date, 14d) | 2 (`include-more-detail-about-detail`, `verify-facts-numbers-dates`) |
| Burst-only rules (single-date, ≥3 hits) | 7 — all ephemeral content |

---

[analyst-autoresearch]
7 changes: 7 additions & 0 deletions Gradata/src/gradata/hooks/adapters/_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,13 @@ def hook_command(brain_dir: Path) -> str:
)


def auto_correct_command(brain_dir: Path) -> str:
return (
f"BRAIN_DIR={shlex.quote(str(brain_dir))} "
f"{shlex.quote(sys.executable)} -m gradata.hooks.auto_correct"
)


def mcp_command(brain_dir: Path) -> list[str]:
return [sys.executable, "-m", "gradata.mcp_server", "--brain-dir", str(brain_dir)]

Expand Down
104 changes: 65 additions & 39 deletions Gradata/src/gradata/hooks/adapters/claude_code.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
WRITE_TOOL_ALIASES,
InstallResult,
_normalize_tool_name,
auto_correct_command,
extract_from_edit_args,
extract_from_write_args,
failure,
Expand Down Expand Up @@ -57,36 +58,58 @@ def install(brain_dir: Path, agent_config_path: Path) -> InstallResult:
sig = hook_signature(AGENT, brain_dir)
data = read_json(agent_config_path)
hooks = data.setdefault("hooks", {})

# PreToolUse: inject_brain_rules
pre_tool = hooks.setdefault("PreToolUse", [])
if any(sig in str(item) for item in pre_tool):
return InstallResult(
AGENT, agent_config_path, "already_present", "hook already present"
pre_present = any(sig in str(item) for item in pre_tool)
if not pre_present:
pre_tool.append(
{
"matcher": "*",
"hooks": [
{
"type": "command",
"command": hook_command(brain_dir),
"id": sig,
}
],
}
)
pre_tool.append(
{
"matcher": "*",
"hooks": [
{
"type": "command",
"command": hook_command(brain_dir),
"id": sig,
}
],
}
)

# PostToolUse: auto_correct (Edit|Write capture)
post_tool = hooks.setdefault("PostToolUse", [])
post_present = any(sig in str(item) for item in post_tool)
if not post_present:
post_tool.append(
{
"matcher": "Edit|Write",
"hooks": [
{
"type": "command",
"command": auto_correct_command(brain_dir),
"id": sig,
}
],
}
)
Comment on lines +79 to +94

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# 1) Inspect HOST_MATRIX expected_events for claude-code in the smoke test
fd -t f 'test_install_smoke_matrix.py' --exec rg -n -C3 'HOST_MATRIX|expected_events|claude-code|PostToolUse|PreToolUse' {}

# 2) Compare matcher shapes between adapter install and generate_hook_config
rg -nP -C3 "matcher" --type=py -g '!**/tests/**' gradata/src/gradata/hooks 2>/dev/null || \
  rg -nP -C3 "matcher" --type=py

Repository: Gradata/gradata

Length of output: 21624


Fix smoke-matrix coverage for Claude Code PostToolUse + clarify matcher semantics

  1. tests/test_install_smoke_matrix.py sets HOST_MATRIX for claude-code to expected_events={"PreToolUse"}; since the test only checks expected_events <= set(hooks), it will not assert the new PostToolUse wiring—so a regression in Gradata/src/gradata/hooks/adapters/claude_code.py won’t fail the smoke test.
  2. claude_code.py installs a single PostToolUse entry with matcher "Edit|Write", while auto_correct.generate_hook_config() emits two separate matchers "Edit" and "Write". Confirm that Claude Code’s matcher evaluation treats "Edit|Write" as intended and does not accidentally trigger on MultiEdit (or add a test to lock this down).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Gradata/src/gradata/hooks/adapters/claude_code.py` around lines 79 - 94, The
smoke test and adapter are out of sync: tests/HOST_MATRIX for claude-code only
expects "PreToolUse" so the new PostToolUse wiring in
Gradata/src/gradata/hooks/adapters/claude_code.py will not be asserted, and the
adapter uses a single matcher "Edit|Write" while
auto_correct.generate_hook_config() emits separate "Edit" and "Write" matchers
(risking accidental matches like "MultiEdit"). Fix by (1) updating the test
HOST_MATRIX entry for "claude-code" to include "PostToolUse" in expected_events
so the smoke matrix will catch regressions, and (2) change the PostToolUse hook
installation in claude_code.py to either add two distinct matcher entries
("Edit" and "Write") or use an exact-match regex like "^(Edit|Write)$" (adjust
the code that appends the dict using auto_correct_command and sig) to match
auto_correct.generate_hook_config() semantics exactly.


write_json(agent_config_path, data)
return InstallResult(AGENT, agent_config_path, "added", "installed PreToolUse hook")
added = (0 if pre_present else 1) + (0 if post_present else 1)
if added == 0:
return InstallResult(
AGENT, agent_config_path, "already_present", "both hooks already present"
)
return InstallResult(AGENT, agent_config_path, "added", f"installed {added} hook(s)")
except Exception as exc:
return failure(AGENT, agent_config_path, exc)


def uninstall(brain_dir: Path, agent_config_path: Path) -> InstallResult:
"""Reverse ``install()``: drop the signature-matching PreToolUse entry.
"""Reverse install(): drop signature-matching PreToolUse + PostToolUse entries.

Idempotent calling on an already-clean config returns ``already_present``
Idempotent -- calling on an already-clean config returns already_present
(semantically: 'already in the desired absent state'). Empty containers
are pruned. User-owned PreToolUse entries (without our signature) are
preserved verbatim.
are pruned. User-owned entries (without our signature) are preserved.
"""
try:
if not agent_config_path.is_file():
Expand All @@ -98,30 +121,33 @@ def uninstall(brain_dir: Path, agent_config_path: Path) -> InstallResult:
hooks = data.get("hooks")
if not isinstance(hooks, dict):
return InstallResult(AGENT, agent_config_path, "already_present", "no hooks block")
pre_tool = hooks.get("PreToolUse")
if not isinstance(pre_tool, list):
return InstallResult(AGENT, agent_config_path, "already_present", "no PreToolUse")

removed = 0
kept: list = []
for entry in pre_tool:
entry_str = str(entry)
if sig in entry_str:
# Either the entry's `hooks[].id` carries our sig, or the
# whole entry was ours. Drop it.
removed += 1

total_removed = 0
for event_key in ("PreToolUse", "PostToolUse"):
entries = hooks.get(event_key)
if not isinstance(entries, list):
continue
kept.append(entry)
if removed == 0:

kept: list = []
for entry in entries:
if sig in str(entry):
total_removed += 1
continue
kept.append(entry)

if kept:
hooks[event_key] = kept
else:
hooks.pop(event_key, None)

if total_removed == 0:
return InstallResult(AGENT, agent_config_path, "already_present", "hook not present")

if kept:
hooks["PreToolUse"] = kept
else:
hooks.pop("PreToolUse", None)
if not hooks:
data.pop("hooks", None)
write_json(agent_config_path, data)
return InstallResult(AGENT, agent_config_path, "removed", f"removed {removed} hook entry")
return InstallResult(
AGENT, agent_config_path, "removed", f"removed {total_removed} hook entry"
)
except Exception as exc:
return failure(AGENT, agent_config_path, exc)