self-improvement through lessons learned

[jeremymanning]

we should add two new CRON jobs:

1. learning from errors: audit status logs and look for recent errors. if there are patterns, suggest an improvement to llmxive that can be planned, implemented, and tested.

2. perusing open issues: audit open issues and see if there is anything of high value to work on. if so, submit it to the llmxive pipeline so that it can be worked on.

these are special projects/jobs in that, whereas projects are typically sandboxed, success in these llmxive-improvement projects should result in changes to llmxive itself.

[jeremymanning]

Deep dive: an autonomous, bounded self-improvement lane

Bottom line

This should not reuse the normal research-project state machine. Existing intake creates brainstormed research projects under projects/PROJ-*, and downstream agents are intentionally scoped there. A self-improvement task needs a separate contract whose successful output is a branch/PR against llmXive itself.

The repository already has most of the observation layer:

• Canonical JSONL activity lives under state/run-log; submission intake writes there too (source).
• state/advance_errors already carries typed fingerprints, classes, recurrence counts, evidence, remediation, and retry disposition (taxonomy, record construction).
• The dashboard already distinguishes expected convergence kickbacks from genuine failures, which should be reused rather than treating every outcome=failed as a defect (source).
• The hourly maintenance lane already performs issue intake with bounded backlog generation and explicit permissions (workflow, steps).

What is missing is a safe promotion path:

observations/issues
      ↓
deterministic evidence + dedupe
      ↓
proposal model
      ↓
independent assessor
      ↓
isolated implementation branch
      ↓
tests + protected eval + replay canary
      ↓
automatic merge through required checks
      ↓
post-merge health canary
      ↓
keep or automatic revert

There should be no mandatory human approval gate. Human attention is an optional escalation after a circuit breaker trips, not part of the happy path.

Relevant prior work

The strongest transferable ideas are:

• Reflexion (NeurIPS 2023): convert environment feedback into linguistic lessons stored as episodic memory. For llmXive, the “environment feedback” should be deterministic failure clusters and test outcomes—not unconstrained model introspection.
• Self-Refine (NeurIPS 2023): iterative feedback/refinement can improve first-pass output, but its generator and critic may be the same model. Here the final verifier should be independent to reduce correlated errors.
• Voyager: combines an automatic curriculum, persistent executable skills, environment feedback, and self-verification. The useful analogue is a persistent archive of attempted improvements and their measured outcomes.
• SWE-agent and SWE-bench (ICLR 2024): repository-level issue resolution needs a purpose-built agent/computer interface and real execution tests; an issue description plus one generated patch is not enough.
• Darwin Gödel Machine: self-modifications are retained only after empirical evaluation, with an archive preserving alternatives and ancestry. llmXive should adopt empirical promotion plus provenance, while keeping a much smaller permitted change surface.
• GitHub recommends least-privilege tokens and explicitly warns that untrusted PR/issue content in privileged workflows is a repository-compromise risk (GitHub secure-use guidance). Required status checks and protected branches provide the right automatic promotion boundary (GitHub protected branches).

Proposed data model

Add src/llmxive/self_improvement/ and keep this state separate from Project/Stage:

class ImprovementCandidate(BaseModel):
    id: str                         # sha256(source_kind + source_key + evidence)
    source_kind: Literal["errors", "issue"]
    source_key: str                 # fingerprint or "issue:123"
    title: str
    evidence: list[EvidenceRef]     # run IDs, project IDs, issue URL, commit SHA
    proposed_change: str
    acceptance_tests: list[str]
    impact: int                     # 1..5
    confidence: int                 # 1..5
    tractability: int               # 1..5
    testability: int                # 1..5
    blast_radius: int               # 1..5, lower is better
    status: Literal[
        "proposed", "verified", "implementing", "canary",
        "merged", "reverted", "rejected", "circuit_broken"
    ]
    proposer_model: str
    verifier_model: str | None
    base_sha: str
    branch: str | None
    pr_number: int | None
    parent_candidate_id: str | None

Persist records under state/self-improvement/<YYYY-MM>/<id>.json. The stable ID is the dedupe key; a second proposal for the same source/evidence updates the existing record rather than spawning another job.

Cron 1: learn from errors

Add .github/workflows/self-improvement-errors.yml, daily at an off-peak minute and manually dispatchable.

1. Read the last 14 days of state/run-log and current state/advance_errors.
2. Reuse a public version of advance_ledger.classify_message; do not invent a second taxonomy.
3. Reuse/extract _activity_severity so expected StagePanelKickback records remain soft mechanics.
4. Avoid double-counting an advance_errors aggregate and its corresponding run-log event. Use:
   • run log for time-bounded event frequency;
   • ledger for current recurrence/status/evidence;
   • distinct affected projects as the primary support measure.
5. Require an evidence floor, e.g. at least three distinct projects, or two projects plus five recent occurrences. Single-project failures may still qualify only if classed as invariant/security/data-loss.
6. Give the proposal agent only compact, redacted evidence; it must emit a structured change hypothesis and executable acceptance criteria.
7. An independent assessor receives the proposal and evidence, but not the proposer’s hidden reasoning. Promote only if:
   • impact, confidence, tractability, and testability are each ≥3/5;
   • blast radius ≤2/5;
   • at least one acceptance test fails on the base revision or a replay fixture demonstrates the defect;
   • the candidate key is not active or inside a 30-day cooldown.

Core collection logic:

for entry in recent_run_log:
    reason = entry.failure_reason or entry.error or ""
    if severity(entry.outcome, reason) != "hard":
        continue
    fingerprint, error_class = classify_message(reason)
    groups[fingerprint].add(
        project_id=entry.project_id,
        run_id=entry.run_id,
        occurred_at=entry.ended_at,
    )

for fingerprint, group in groups.items():
    ledger = matching_current_ledger_records(fingerprint)
    if group.distinct_projects >= 3 or is_critical(error_class, group):
        emit_candidate(
            source_key=f"error:{fingerprint}",
            evidence=group.refs + ledger.refs,
        )

Cron 2: inspect open issues

Add .github/workflows/self-improvement-issues.yml, daily at a different off-peak minute.

Use a two-stage ranking process:

1. Deterministic filter:
   • exclude bot-authored issues and human-submission/new-paper;
   • exclude this meta issue itself;
   • exclude issues with an active candidate/PR or cooldown record;
   • exclude issues already satisfied by current main;
   • cap input body/comment size and treat all issue text as untrusted data.
2. Cheap deterministic rank by maintainer labels, recency, evidence/reproduction, affected surface, existing discussion, and estimated scope.
3. Let the proposer assess only the top 10.
4. Let an independent verifier rerank the top three with a fixed rubric: expected_value, confidence, tractability, testability, blast_radius, and duplicate_probability.
5. Autonomously enqueue at most one issue per day above the threshold.

Do not route these through _create_brainstormed_project: that helper explicitly creates a research Project at Stage.BRAINSTORMED under projects/ (source). Instead create an ImprovementCandidate, link the source issue, and mark it with a machine-readable comment/label such as self-improvement:claimed.

Autonomous implementation and promotion

A third reusable execution workflow can drain candidates created by either cron. It need not itself be scheduled; dispatch from the discovery jobs is sufficient.

1. Create bot/self-improvement/<candidate-id> from the recorded base_sha.
2. Run the implementer in an isolated checkout/container with:
   • no production secrets;
   • no issue text interpolated into shell;
   • outbound network disabled unless the plan explicitly requires an allowlisted endpoint;
   • wall-clock and model-credit budgets.
3. Enforce a default diff policy:
   • ≤8 files and ≤300 changed lines;
   • allow src/llmxive/**, tests/**, and narrowly relevant docs;
   • reject edits to .github/workflows/**, secrets/auth, branch protection, the self-improvement policy/eval baseline, dependency locks, or generated project/state corpora;
   • a separate, more stringent path can later handle infrastructure changes.
4. Run formatter/type checks, targeted tests, full unit suite, preflight, and a protected replay corpus.
5. Run an independent verifier with a different model/backend and read-only tools. It checks issue satisfaction, diff scope, tests, security, and whether tests genuinely exercise the claimed defect.
6. Open a PR and let branch protection auto-merge only after all required checks pass. Use a GitHub App identity for merge/promotion so the patch-producing identity cannot bless its own check.
7. Permit at most one autonomous merge per 24 hours and one active candidate at a time.

Branch protection is the automatic gate—not human approval. Required checks should include:

self-improvement/policy
self-improvement/targeted-regression
self-improvement/full-unit
self-improvement/replay-canary
self-improvement/independent-verifier

The protected replay tests and policy code must not be modifiable by the candidate branch. Otherwise the system can “improve” by weakening its evaluator.

Canary, rollback, and spiral prevention

Pre-merge canary

• Replay the last 7–14 days of relevant state in a temporary checkout.
• Compare candidate vs. base on:
  • target fingerprint frequency;
  • total hard-failure rate;
  • successful stage advances;
  • runtime and paid-model cost;
  • invariant violations;
  • unrelated test/eval regressions.
• Require target improvement and zero material regression. “Tests pass” alone is insufficient.

Post-merge canary

• Record the merge SHA and base SHA.
• Observe the next three relevant cron ticks or 24 hours.
• Automatically create and merge a revert if:
  • target failure rate worsens;
  • all hard failures rise >20%;
  • an invariant/data-loss/security signal appears;
  • two consecutive pipeline workflows fail;
  • cost/runtime exceeds its budget by >25%.

Circuit breakers

• Two automatic reverts in seven days disable promotion while discovery continues.
• Three failed attempts for one candidate move it into a 30-day cooldown.
• Never let a candidate modify the discovery/promoter/evaluator in the same generation that relies on it.
• Preserve every proposal, patch, verification result, canary measurement, merge, and revert as immutable provenance.
• A human may inspect or re-enable after a breaker, but routine progress does not wait for approval.

Staged implementation plan

Phase 1 — observation and proposal, dry-run only

• Add models/store/dedupe.
• Extract public failure-severity logic.
• Implement error mining and issue filtering/ranking.
• Add prompts and independent structured assessor.
• Add both cron workflows with read-only contents and issue-write only where comments/labels are required.
• Persist candidates and post a concise evidence comment, but do not implement.

Tests:

• JSONL corruption and missing-field tolerance.
• Soft kickbacks excluded.
• Ledger/run-log overlap not double-counted.
• Stable candidate IDs and cooldown behavior.
• Automated paper issues excluded.
• Prompt-injection strings remain inert data.
• Pagination and API failure are fail-closed.

Phase 2 — bounded implementation branches

• Add isolated branch/worktree executor.
• Add diff allowlist/size policy and budgets.
• Add targeted test synthesis plus full suite.
• Open draft PRs automatically.
• Do not merge until protected replay/verification is complete.

Phase 3 — autonomous promotion

• Add independent verifier identity/model.
• Configure required checks and GitHub App auto-merge.
• Add pre-merge replay canary.
• Enable one merge/day.

Phase 4 — post-merge learning

• Add health monitoring, automatic revert, cooldowns, and global circuit breaker.
• Feed verified outcomes—not the proposer’s narrative—back into future ranking.
• Track precision: proposed, attempted, merged, retained after 7/30 days, reverted, and target-metric improvement.

Directly verified prototype

I built a read-only prototype in /tmp that:

• scans actual state/run-log and state/advance_errors;
• excludes expected StagePanelKickback;
• groups known failure signatures;
• requires multiple affected projects;
• creates a stable proposal key;
• fetches current open issues;
• excludes automated paper intake;
• ranks the remaining backlog.

Exact command:

PYTHONPATH=src python /tmp/llmxive1242_demo.py \
  --root /Users/jmanning/llmXive \
  --days 14 \
  --top 5

It exited 0 and found these current evidence clusters:

unreachable_reference   39 projects
backend_chain_failed     7 projects
template_rejection       6 projects
StagePanelEscalation     5 projects
filesystem_file_exists   4 projects

It also correctly removed bot-created human-submission/new-paper issues from the open-issue candidates.

One prototype caveat informed the plan above: summing advance_errors.consecutive_count alongside JSONL events overstates recent event counts because the ledger is cumulative. The production version should report distinct recent run-log events separately from ledger retry pressure and deduplicate overlapping evidence. This is exactly why deterministic aggregation and explicit provenance should precede the LLM proposal step.

Acceptance criteria

• Both cron jobs are idempotent and bounded.
• Expected pipeline mechanics do not generate improvement proposals.
• Every candidate has reproducible evidence, a stable dedupe key, acceptance tests, base SHA, proposer/verifier identities, and a budget.
• Automated submissions and already-claimed issues cannot recursively refill the queue.
• Candidate code cannot alter its own protected policy/eval gate.
• No patch reaches main unless targeted tests, the full suite, replay canary, policy check, and independent verifier pass.
• Successful candidates auto-merge without mandatory human approval.
• A harmful retained change is automatically reverted, and repeated failures trip a global circuit breaker.
• Every proposal, decision, patch, result, merge, and revert is auditable.