Skip to content

[FEATURE] Closed-PR postmortem miner #578

Description

@acipat

Feature Description

Problem Statement

Many agent-generated PRs are rejected for predictable reasons: wrong premise, effort underestimated, change-detector tests, or conflicts with upstream design intent. The pipeline creates issues and PRs but does not systematically mine closed/rejected PRs for patterns that would prevent future failures — the same failure modes recur across cycles because the historical signal is never turned into durable rules.

Proposed Solution

Add a closed-PR postmortem miner that:

  1. Periodically scans closed/merged/rejected PRs in the fork (gh pr list --state closed).
  2. Classifies the close reason by reading the closing comment and labels (e.g., rejected, needs-work, implemented-on-main, duplicate).
  3. Emits durable rules or checklists for the research/analysis stages — e.g., "check for change-detector tests when touching model metadata," "decompose any issue touching provider configs."
  4. Feeds these rules back into the analysis skill as advisory pre-flight checks.

This turns historical failure data into an improvement signal — the central self-evolution mechanism. It is lightweight because it only reads GitHub metadata and existing close comments.

Value Proposition

  • Impact: 0.7
  • Effort: 0.4
  • Priority Score: 1.18

Research Evidence

  • Software engineering literature on postmortems and defect prediction
  • AutoGen/OpenHands community feedback loops on rejected PRs

Implementation Plan

  1. New skill evolution-postmortem-miner + cron job.
  2. Step 1: fetch closed PRs since last run (bounded window).
  3. Step 2: classify close reason from labels + closing comment.
  4. Step 3: deduplicate against existing rules.
  5. Step 4: emit new rules as structured checklist entries in the analysis skill's config.
  6. Output: a rules JSON file consumed by the analysis stage.

Success Criteria

  • Closed PRs are classified by close reason automatically.
  • Recurring failure patterns produce durable analysis-stage rules.
  • The number of PRs rejected for the same reason decreases over subsequent cycles.

Metadata

Metadata

Assignees

No one assigned

    Labels

    rejectedNot accepted by evolution — see closing comment

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions