feat(PAIUpgrade): mandatory cross-recommendation interference analysis (Step 6b)#1340
Open
82ATW wants to merge 1 commit into
Open
feat(PAIUpgrade): mandatory cross-recommendation interference analysis (Step 6b)#134082ATW wants to merge 1 commit into
82ATW wants to merge 1 commit into
Conversation
…lysis (Step 6b) Prior-Status gating checks each recommendation against prior work; nothing checked recommendations against EACH OTHER. On large reports this lets measurement confounds, same-file collisions, input dependencies, and doctrine-bundling problems ship undetected until implementation. Adds: - Workflows/Upgrade.md: Step 6b — four-class interference matrix + mandatory adversarial verification by a second agent, gated at >10 recommendations (<=10 stays a one-liner; quick modes unaffected) - References/OutputFormat.md: 'Interactions & Sequencing' as canonical section 4 + section spec + hard rule 11 (every conflict carries a guard) - SKILL.md: extraction rule 4b, updated section order, origin gotcha Origin: a 44-recommendation report shipped fully Prior-Status-gated with zero pairwise analysis; the retrofit found two high-severity eval- contaminating confounds and six same-file collisions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
PAIUpgrade's synthesis gates every recommendation against prior work (Thread 0 → Prior Status), but nothing checks recommendations against each other. On large reports, that lets four classes of problems ship undetected until implementation time: measurement confounds (one rec changes the thing another rec measures), same-file collisions, input dependencies (one rec evicts what another reads), and doctrine/version bundling (several recs editing one versioned artifact).
What this adds
Four files under
Releases/v5.0.0/.claude/skills/PAIUpgrade/:Workflows/Upgrade.md— new Step 6b: a four-class interference matrix plus a mandatory adversarial verification pass by a second agent (different model family when available; fresh-context same-model fallback otherwise — disputes resolve by probing ground truth, never by model preference). Gated at >10 recommendations (~45 pairs — roughly where a reliable mental pairwise sweep ends); at ≤10 the section reduces to a single line so quick modes stay fast. Every identified conflict must carry a guard or the report is blocked.References/OutputFormat.md—⚠️ Interactions & Sequencingjoins the canonical section order (position 4), with the section spec and hard rule 11: Prior-Status gates against the past; this gates against each other.SKILL.md— extraction sub-rule under rule 4, updated section-order summary, and a Gotchas entry.References/ExampleReport.md— the worked example now demonstrates the ≤10 one-liner form, staying consistent with the spec it illustrates.Origin
Found in practice: a 44-recommendation report shipped fully Prior-Status-gated with zero pairwise analysis; the retrofit interference pass found two high-severity measurement confounds (a fallback setting and a safety-routing behavior each silently swapping the model under an A/B eval) plus six same-file collisions. The patch itself was hardened through the same adversarial-review loop it mandates.
Scope notes
Releases/v5.0.0tree (the shipped runtime). Happy to also port toPacks/Utilities/src/PAIUpgradeif that's the preferred source of truth — that copy currently reflects an earlier generation of the workflow.Workflows/AlgorithmUpgrade.md(a sibling recommendation-emitting workflow) is intentionally not gated here to keep the change minimal; happy to follow up if useful.