refactor(.agent): host-agent review protocol + structural fixes

Written by Minimax-M2.7 in Claude Code Harness.

Shifts the reasoning boundary: Python handles mechanical filing
(cluster, extract, stage, heuristic prefilter, decay, archive).
The host agent (Claude Code, Codex, Cursor, Windsurf) handles
validation via CLI tools using the LLM it already has. The brain
is no longer coupled to a specific provider SDK.

Key modules
- memory/auto_dream.py: staging-only dream cycle. No validation,
  no graduation, no git commit. Safe on Stop hooks or cron.
- memory/validate.py: heuristic prefilter (length + exact
  duplicate against accepted lessons). Deterministic, no LLM.
- memory/review_state.py: candidate lifecycle (staged/provisional/
  accepted/rejected/superseded) + append-only decision log.
  Stamps evidence + lessons hash at decision time so re-stage
  only fires on real change, not churn.
- memory/render_lessons.py: lessons.jsonl as source of truth;
  LESSONS.md rendered from it with sentinel-preserved user
  content. Auto-migrates legacy bullets on first run; dedupes
  by lesson id (handles provisional -> accepted transitions).
- memory/cluster.py: proper single-linkage Jaccard clustering
  with bridge-merge; claim+conditions-stable pattern ids.
- memory/promote.py: re-stage gate keyed on NEW evidence and
  specific-blocker presence, not reviewer identity or whole-file
  hashes. Decay-only evidence shrinkage doesn't churn. Full
  lifecycle respects all three subdirs (staged/rejected/graduated).
- harness/context_budget.py: query-aware retrieval filtered to
  status=accepted only; loads AGENTS.md, DECISIONS.md,
  REVIEW_QUEUE.md per the stated read order.
- harness/llm.py: used only by standalone conductor path; the
  memory layer does not import it.
- harness/text.py: shared word_set + jaccard.
- harness/hooks/_provenance.py: source metadata on every
  episodic entry (confidence, source{skill, run_id, commit_sha},
  evidence_ids).
- tools/{list_candidates,graduate,reject,reopen}.py: host-agent
  CLI. graduate.py requires --rationale and is atomic: lessons
  write first, candidate move last. --supersedes exempts the
  target lesson from the duplicate check.

Workflow notes
- Same pattern across cluster membership changes keeps the same
  id and lifecycle history.
- Accepted lessons are terminal. Provisional acceptances re-enter
  review when new evidence arrives.
- Rejected candidates re-enter review only when evidence grows
  OR the specific blocking lesson disappears.
- Provisional supersessions do not strike the active lesson
  until the superseder is itself accepted.
- Retrieval filters to accepted lessons only; empty filter
  returns empty rather than leaking raw markdown.

Not tracked (stays per-user)
- memory/personal/PREFERENCES.md
- memory/working/* (WORKSPACE.md, REVIEW_QUEUE.md)
- memory/episodic/AGENT_LEARNINGS.jsonl
- memory/candidates/ (transient)
- memory/semantic/lessons.jsonl (grows from graduations)

Author: codejunkie99

.agent/AGENTS.md

@@ -7,10 +7,34 @@ same memory, skills, and protocols.
 ## Memory (read in this order)
 - `memory/personal/PREFERENCES.md` — stable user conventions
 - `memory/working/WORKSPACE.md` — current task state
-- `memory/semantic/LESSONS.md` — distilled patterns, read before decisions
+- `memory/working/REVIEW_QUEUE.md` — pending candidate lessons waiting for you
 - `memory/semantic/DECISIONS.md` — past architectural choices
+- `memory/semantic/LESSONS.md` — distilled patterns (rendered from `lessons.jsonl`)
 - `memory/episodic/AGENT_LEARNINGS.jsonl` — raw experience log (top-k by salience)
 
+## Review Queue (host-agent responsibility)
+
+Candidate lessons are clustered + staged automatically by `memory/auto_dream.py`.
+The host agent — you — does the actual review using the CLI tools below.
+
+Check `memory/working/REVIEW_QUEUE.md` at session start. If pending > 10 or
+oldest staged > 7 days, review before substantive work.
+
+Workflow:
+1. `python .agent/tools/list_candidates.py` — pending candidates, sorted by priority
+2. For each: decide accept / reject / defer based on claim, evidence_ids,
+   cluster_size, and any contradictions with existing LESSONS.md
+3. `python .agent/tools/graduate.py <id> --rationale "..."` to accept
+4. `python .agent/tools/reject.py <id> --reason "..."` to reject
+5. `python .agent/tools/reopen.py <id>` to requeue a previously-rejected item
+6. Review in a **batch**, not one-by-one — cross-candidate contradictions
+   only surface when you see multiple at once.
+
+The heuristic prefilter in `memory/validate.py` has already dropped obvious
+junk (too-short claims, exact duplicates). Everything staged needs real
+judgment. Rationale is required for graduation — rubber-stamped promotions
+are the exact failure mode this layer prevents.
+
 ## Skills
 - `skills/_index.md` — read first for discovery
 - `skills/_manifest.jsonl` — machine-readable skill metadata
@@ -24,8 +48,12 @@ same memory, skills, and protocols.
 
 ## Rules
 1. Check memory before decisions you have been corrected on before.
-2. Log every significant action to `memory/episodic/AGENT_LEARNINGS.jsonl`.
-3. Update `memory/working/WORKSPACE.md` as you work; archive on completion.
-4. Follow `protocols/permissions.md` strictly. Blocked means blocked.
-5. When a self-rewrite hook fires, propose conservative edits only.
-6. The harness is dumb on purpose. Reasoning lives in skills and memory.
+2. If `REVIEW_QUEUE.md` shows backlog past threshold, handle it before the new task.
+3. Log every significant action to `memory/episodic/AGENT_LEARNINGS.jsonl`
+   via `.agent/tools/memory_reflect.py`.
+4. Update `memory/working/WORKSPACE.md` as you work; archive on completion.
+5. Never hand-edit `memory/semantic/LESSONS.md` — it's rendered from
+   `lessons.jsonl`. Use `graduate.py` / `reject.py` instead.
+6. Follow `protocols/permissions.md`. Blocked means blocked.
+7. When a self-rewrite hook fires, propose conservative edits only.
+8. The harness is dumb on purpose. Reasoning lives in skills + the host agent.

.agent/harness/conductor.py

@@ -2,35 +2,12 @@
 import os, sys
 from context_budget import build_context
 from hooks.post_execution import log_execution
+from llm import call_model
 
 RESERVED = 40000
 MAX_CTX = int(os.getenv("AGENT_MAX_CONTEXT", "128000"))
 
 
-def _call_model(system, user):
-    provider = os.getenv("AGENT_PROVIDER", "anthropic").lower()
-    if provider == "anthropic":
-        from anthropic import Anthropic
-        c = Anthropic()
-        r = c.messages.create(
-            model=os.getenv("AGENT_MODEL", "claude-sonnet-4-5"),
-            max_tokens=4096, temperature=0.3,
-            system=system,
-            messages=[{"role": "user", "content": user}],
-        )
-        return r.content[0].text
-    if provider == "openai":
-        from openai import OpenAI
-        c = OpenAI()
-        r = c.chat.completions.create(
-            model=os.getenv("AGENT_MODEL", "gpt-4o"),
-            messages=[{"role": "system", "content": system},
-                      {"role": "user", "content": user}],
-        )
-        return r.choices[0].message.content
-    raise ValueError(f"unknown provider: {provider}")
-
-
 SYSTEM_PREAMBLE = (
     "You are an agent with externalized memory, skills, and protocols.\n"
     "Your memory, skills, and constraints are in the context below.\n"
@@ -43,7 +20,7 @@ def run(user_input: str) -> str:
     context, used = build_context(user_input, budget=MAX_CTX - RESERVED)
     system = SYSTEM_PREAMBLE + context
     try:
-        result = _call_model(system, user_input)
+        result = call_model(system, user_input)
         log_execution("conductor", user_input[:100], result[:500], True)
         return result
     except Exception as e:

.agent/harness/context_budget.py

@@ -1,8 +1,23 @@
-"""Assemble context from memory + matched skills + protocols within a token budget."""
-import json, os
+"""Assemble context from memory + matched skills + protocols within a token budget.
+
+Query-aware: episodes and lessons are scored against user_input so the agent
+sees the memory that matters for *this* task, not just the most salient memory
+in general. Always-on slots (PREFERENCES, WORKSPACE, permissions) are loaded
+whole regardless of query — they're cheap and safety-critical.
+"""
+import json, os, re, sys
 from salience import salience_score
+from text import word_set, jaccard
 
 ROOT = os.path.join(os.path.dirname(__file__), "..")
+# skill_loader lives in tools/ — make it importable without requiring callers
+# to configure PYTHONPATH themselves
+sys.path.insert(0, os.path.join(ROOT, "tools"))
+RELEVANCE_FLOOR = 0.3  # even zero-overlap episodes surface if very salient
+
+# Keep in sync with memory/validate._extract_lesson_lines — both filters
+# want TERMINAL-only lesson content.
+_STATUS_RE = re.compile(r"status=(\w+)")
 
 
 def _read(path, limit=None):
@@ -13,11 +28,22 @@ def _read(path, limit=None):
     return content[:limit] if limit else content
 
 
-def _tokens(text):
+def _token_estimate(text):
+    """Rough chars-to-tokens estimate for budgeting."""
     return len(text) // 4
 
 
-def _top_episodes(k=5):
+def _relevance(entry_text, query_words):
+    """Fraction of query words that appear in entry. 1.0 when no query."""
+    if not query_words:
+        return 1.0
+    ew = word_set(entry_text)
+    if not ew:
+        return 0.0
+    return len(query_words & ew) / len(query_words)
+
+
+def _top_episodes(query, k=5):
     path = os.path.join(ROOT, "memory/episodic/AGENT_LEARNINGS.jsonl")
     if not os.path.exists(path):
         return ""
@@ -30,7 +56,19 @@ def _top_episodes(k=5):
             entries.append(json.loads(line))
         except json.JSONDecodeError:
             continue
-    entries.sort(key=salience_score, reverse=True)
+
+    query_words = word_set(query)
+
+    def _score(e):
+        text = " ".join([
+            e.get("action", ""),
+            e.get("reflection", ""),
+            e.get("detail", ""),
+        ])
+        rel = _relevance(text, query_words)
+        return salience_score(e) * (RELEVANCE_FLOOR + (1.0 - RELEVANCE_FLOOR) * rel)
+
+    entries.sort(key=_score, reverse=True)
     top = entries[:k]
     return "\n".join(
         f"- [{e.get('timestamp','')[:10]}] {e.get('action','')}: "
@@ -39,38 +77,104 @@ def _top_episodes(k=5):
     )
 
 
+def _lines_up_to_budget(lines, char_budget):
+    out, used = [], 0
+    for line in lines:
+        block = f"- {line}\n"
+        if used + len(block) > char_budget:
+            break
+        out.append(block)
+        used += len(block)
+    return "".join(out)
+
+
+def _top_lessons(query, lessons_md, char_budget=8000):
+    """Rank accepted lesson bullets by query overlap; fall back to original order.
+
+    Only terminal (status=accepted) lessons reach the host agent as retrievable
+    guidance. Provisional, legacy, and superseded bullets exist in LESSONS.md
+    for audit but must not be injected into the system prompt — they'd let the
+    agent act on probationary or stale memory.
+    """
+    lines = []
+    for line in (lessons_md or "").splitlines():
+        s = line.strip()
+        if not s.startswith("- ") or len(s) <= 2:
+            continue
+        # Primary status filter: HTML annotation
+        if "<!--" in s:
+            ann = s.split("<!--", 1)[1]
+            m = _STATUS_RE.search(ann)
+            if m and m.group(1) != "accepted":
+                continue
+        text = s[2:].split("<!--")[0].strip()
+        # Fallback: visual markers
+        if text.startswith("[PROVISIONAL]"):
+            continue
+        if text.startswith("~~") and text.endswith("~~"):
+            continue
+        if text:
+            lines.append(text)
+    if not lines:
+        # No accepted lessons → return empty. Returning raw markdown would
+        # leak the non-terminal content the filter is designed to block.
+        return ""
+
+    query_words = word_set(query)
+    if not query_words:
+        return _lines_up_to_budget(lines, char_budget)
+
+    scored = [(len(query_words & word_set(l)), i, l) for i, l in enumerate(lines)]
+    relevant = sorted([s for s in scored if s[0] > 0], key=lambda s: (-s[0], s[1]))
+
+    if not relevant:
+        return _lines_up_to_budget(lines, char_budget)
+    return _lines_up_to_budget([l for _, _, l in relevant], char_budget)
+
+
 def build_context(user_input: str, budget: int = 88000):
-    """Returns (context_string, tokens_used). Lean by design."""
-    from skill_loader import progressive_load  # lazy to avoid cycles
+    """Returns (context_string, tokens_used). Lean and query-aware."""
     parts, used = [], 0
 
-    # always load: personal preferences + live workspace
-    for rel in ("memory/personal/PREFERENCES.md", "memory/working/WORKSPACE.md"):
+    # always load: personal preferences + live workspace + AGENTS map + DECISIONS
+    # AGENTS.md and DECISIONS.md were missing despite AGENTS.md specifying the
+    # read order — the standalone path was not faithful to its own contract.
+    for rel in (
+        "AGENTS.md",
+        "memory/personal/PREFERENCES.md",
+        "memory/working/WORKSPACE.md",
+        "memory/working/REVIEW_QUEUE.md",
+        "memory/semantic/DECISIONS.md",
+    ):
         text = _read(rel)
         if text:
             parts.append(f"# {rel}\n{text}")
-            used += _tokens(text)
+            used += _token_estimate(text)
 
-    # semantic lessons, truncated
-    lessons = _read("memory/semantic/LESSONS.md", limit=8000)
-    if lessons:
-        parts.append(f"# LESSONS\n{lessons}")
-        used += _tokens(lessons)
+    # query-aware lessons
+    lessons_raw = _read("memory/semantic/LESSONS.md")
+    if lessons_raw:
+        lessons = _top_lessons(user_input, lessons_raw, char_budget=8000)
+        if lessons:
+            parts.append(f"# LESSONS (query-relevant)\n{lessons}")
+            used += _token_estimate(lessons)
 
-    # top episodic by salience
-    episodes = _top_episodes(k=5)
+    # query-aware top episodes
+    episodes = _top_episodes(user_input, k=5)
     if episodes:
-        parts.append(f"# RECENT EPISODES (top by salience)\n{episodes}")
-        used += _tokens(episodes)
+        parts.append(f"# RECENT EPISODES (salience x relevance)\n{episodes}")
+        used += _token_estimate(episodes)
 
-    # matched skills only
+    # matched skills only (progressive_load is already input-matched).
+    # Lazy import so a missing skill_loader doesn't kill context assembly.
     try:
+        from skill_loader import progressive_load
         skills = progressive_load(user_input)
     except Exception:
         skills = []
     for s in skills:
         block = f"## Skill: {s['name']}\n{s['content']}"
-        t = _tokens(block)
+        t = _token_estimate(block)
         if used + t < budget:
             parts.append(block)
             used += t
@@ -79,6 +183,6 @@ def build_context(user_input: str, budget: int = 88000):
     perms = _read("protocols/permissions.md")
     if perms:
         parts.append(f"# PERMISSIONS\n{perms}")
-        used += _tokens(perms)
+        used += _token_estimate(perms)
 
     return "\n\n---\n\n".join(parts), used

.agent/harness/hooks/_provenance.py

@@ -0,0 +1,33 @@
+"""Shared provenance helpers for episodic entries. Cached per-process."""
+import os, subprocess
+
+AGENT_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
+
+_CACHED_COMMIT = None
+_CACHED_RUN_ID = None
+
+
+def run_id():
+    global _CACHED_RUN_ID
+    if _CACHED_RUN_ID is None:
+        _CACHED_RUN_ID = os.environ.get("AGENT_RUN_ID", f"pid-{os.getpid()}")
+    return _CACHED_RUN_ID
+
+
+def commit_sha():
+    global _CACHED_COMMIT
+    if _CACHED_COMMIT is None:
+        try:
+            out = subprocess.run(
+                ["git", "rev-parse", "HEAD"],
+                capture_output=True, text=True, timeout=2,
+                cwd=AGENT_ROOT,
+            )
+            _CACHED_COMMIT = out.stdout.strip() if out.returncode == 0 else ""
+        except Exception:
+            _CACHED_COMMIT = ""
+    return _CACHED_COMMIT
+
+
+def build_source(skill):
+    return {"skill": skill, "run_id": run_id(), "commit_sha": commit_sha()}

.agent/harness/hooks/on_failure.py

@@ -1,5 +1,6 @@
 """Failures are learning. High pain score + rewrite flag after repeat offenses."""
 import json, datetime, os
+from ._provenance import build_source
 
 ROOT = os.path.join(os.path.dirname(__file__), "..", "..")
 EPISODIC = os.path.join(ROOT, "memory/episodic/AGENT_LEARNINGS.jsonl")
@@ -30,7 +31,8 @@ def _count_recent_failures(skill_name):
     return count
 
 
-def on_failure(skill_name, action, error, context=""):
+def on_failure(skill_name, action, error, context="", confidence=0.9,
+               evidence_ids=None):
     entry = {
         "timestamp": datetime.datetime.now().isoformat(),
         "skill": skill_name,
@@ -42,8 +44,13 @@ def on_failure(skill_name, action, error, context=""):
         "reflection": f"FAILURE in {skill_name}: {type(error).__name__}: "
                       f"{str(error)[:200]}",
         "context": context[:300],
+        "confidence": confidence,
+        "source": build_source(skill_name),
+        "evidence_ids": list(evidence_ids) if evidence_ids else [],
     }
-    recent = _count_recent_failures(skill_name)
+    # _count_recent_failures returns PRIOR failures only; add 1 for this one
+    # so the rewrite flag fires on the Nth failure, not the (N+1)th.
+    recent = _count_recent_failures(skill_name) + 1
     if recent >= FAILURE_THRESHOLD:
         entry["reflection"] += (
             f" | THIS SKILL HAS FAILED {recent} TIMES IN {WINDOW_DAYS}d. "

.agent/harness/hooks/post_execution.py

@@ -1,12 +1,13 @@
 """Runs after every action. Appends a structured entry to episodic memory."""
 import json, datetime, os
+from ._provenance import build_source
 
 ROOT = os.path.join(os.path.dirname(__file__), "..", "..")
 EPISODIC = os.path.join(ROOT, "memory/episodic/AGENT_LEARNINGS.jsonl")
 
 
 def log_execution(skill_name, action, result, success, reflection="",
-                  importance=5):
+                  importance=5, confidence=0.5, evidence_ids=None):
     os.makedirs(os.path.dirname(EPISODIC), exist_ok=True)
     entry = {
         "timestamp": datetime.datetime.now().isoformat(),
@@ -17,6 +18,9 @@ def log_execution(skill_name, action, result, success, reflection="",
         "pain_score": 2 if success else 7,
         "importance": importance,
         "reflection": reflection,
+        "confidence": confidence,
+        "source": build_source(skill_name),
+        "evidence_ids": list(evidence_ids) if evidence_ids else [],
     }
     with open(EPISODIC, "a") as f:
         f.write(json.dumps(entry) + "\n")