|
| 1 | +# GPT-5.4 / Codex Parity Maintainer Notes |
| 2 | + |
| 3 | +This note explains how to review the GPT-5.4 / Codex parity program as four merge units without losing the original six-contract architecture. |
| 4 | + |
| 5 | +## Merge units |
| 6 | + |
| 7 | +### PR A: strict-agentic execution |
| 8 | + |
| 9 | +Owns: |
| 10 | + |
| 11 | +- `executionContract` |
| 12 | +- GPT-5-first same-turn follow-through |
| 13 | +- `update_plan` as non-terminal progress tracking |
| 14 | +- explicit blocked states instead of plan-only silent stops |
| 15 | + |
| 16 | +Does not own: |
| 17 | + |
| 18 | +- auth/runtime failure classification |
| 19 | +- permission truthfulness |
| 20 | +- replay/continuation redesign |
| 21 | +- parity benchmarking |
| 22 | + |
| 23 | +### PR B: runtime truthfulness |
| 24 | + |
| 25 | +Owns: |
| 26 | + |
| 27 | +- Codex OAuth scope correctness |
| 28 | +- typed provider/runtime failure classification |
| 29 | +- truthful `/elevated full` availability and blocked reasons |
| 30 | + |
| 31 | +Does not own: |
| 32 | + |
| 33 | +- tool schema normalization |
| 34 | +- replay/liveness state |
| 35 | +- benchmark gating |
| 36 | + |
| 37 | +### PR C: execution correctness |
| 38 | + |
| 39 | +Owns: |
| 40 | + |
| 41 | +- provider-owned OpenAI/Codex tool compatibility |
| 42 | +- parameter-free strict schema handling |
| 43 | +- replay-invalid surfacing |
| 44 | +- paused, blocked, and abandoned long-task state visibility |
| 45 | + |
| 46 | +Does not own: |
| 47 | + |
| 48 | +- self-elected continuation |
| 49 | +- generic Codex dialect behavior outside provider hooks |
| 50 | +- benchmark gating |
| 51 | + |
| 52 | +### PR D: parity harness |
| 53 | + |
| 54 | +Owns: |
| 55 | + |
| 56 | +- first-wave GPT-5.4 vs Opus 4.6 scenario pack |
| 57 | +- parity documentation |
| 58 | +- parity report and release-gate mechanics |
| 59 | + |
| 60 | +Does not own: |
| 61 | + |
| 62 | +- runtime behavior changes outside QA-lab |
| 63 | +- auth/proxy/DNS simulation inside the harness |
| 64 | + |
| 65 | +## Mapping back to the original six contracts |
| 66 | + |
| 67 | +| Original contract | Merge unit | |
| 68 | +| ---------------------------------------- | ---------- | |
| 69 | +| Provider transport/auth correctness | PR B | |
| 70 | +| Tool contract/schema compatibility | PR C | |
| 71 | +| Same-turn execution | PR A | |
| 72 | +| Permission truthfulness | PR B | |
| 73 | +| Replay/continuation/liveness correctness | PR C | |
| 74 | +| Benchmark/release gate | PR D | |
| 75 | + |
| 76 | +## Review order |
| 77 | + |
| 78 | +1. PR A |
| 79 | +2. PR B |
| 80 | +3. PR C |
| 81 | +4. PR D |
| 82 | + |
| 83 | +PR D is the proof layer. It should not be the reason runtime-correctness PRs are delayed. |
| 84 | + |
| 85 | +## What to look for |
| 86 | + |
| 87 | +### PR A |
| 88 | + |
| 89 | +- GPT-5 runs act or fail closed instead of stopping at commentary |
| 90 | +- `update_plan` no longer looks like progress by itself |
| 91 | +- behavior stays GPT-5-first and embedded-Pi scoped |
| 92 | + |
| 93 | +### PR B |
| 94 | + |
| 95 | +- auth/proxy/runtime failures stop collapsing into generic “model failed” handling |
| 96 | +- `/elevated full` is only described as available when it is actually available |
| 97 | +- blocked reasons are visible to both the model and the user-facing runtime |
| 98 | + |
| 99 | +### PR C |
| 100 | + |
| 101 | +- strict OpenAI/Codex tool registration behaves predictably |
| 102 | +- parameter-free tools do not fail strict schema checks |
| 103 | +- replay and compaction outcomes preserve truthful liveness state |
| 104 | + |
| 105 | +### PR D |
| 106 | + |
| 107 | +- the scenario pack is understandable and reproducible |
| 108 | +- the pack includes a mutating replay-safety lane, not only read-only flows |
| 109 | +- reports are readable by humans and automation |
| 110 | +- parity claims are evidence-backed, not anecdotal |
| 111 | + |
| 112 | +Expected artifacts from PR D: |
| 113 | + |
| 114 | +- `qa-suite-report.md` / `qa-suite-summary.json` for each model run |
| 115 | +- `qa-agentic-parity-report.md` with aggregate and scenario-level comparison |
| 116 | +- `qa-agentic-parity-summary.json` with a machine-readable verdict |
| 117 | + |
| 118 | +## Release gate |
| 119 | + |
| 120 | +Do not claim GPT-5.4 parity or superiority over Opus 4.6 until: |
| 121 | + |
| 122 | +- PR A, PR B, and PR C are merged |
| 123 | +- PR D runs the first-wave parity pack cleanly |
| 124 | +- runtime-truthfulness regression suites remain green |
| 125 | +- the parity report shows no fake-success cases and no regression in stop behavior |
| 126 | + |
| 127 | +```mermaid |
| 128 | +flowchart LR |
| 129 | + A["PR A-C merged"] --> B["Run GPT-5.4 parity pack"] |
| 130 | + A --> C["Run Opus 4.6 parity pack"] |
| 131 | + B --> D["qa-suite-summary.json"] |
| 132 | + C --> E["qa-suite-summary.json"] |
| 133 | + D --> F["qa parity-report"] |
| 134 | + E --> F |
| 135 | + F --> G["Markdown report + JSON verdict"] |
| 136 | + G --> H{"Pass?"} |
| 137 | + H -- "yes" --> I["Parity claim allowed"] |
| 138 | + H -- "no" --> J["Keep runtime fixes / review loop open"] |
| 139 | +``` |
| 140 | + |
| 141 | +The parity harness is not the only evidence source. Keep this split explicit in review: |
| 142 | + |
| 143 | +- PR D owns the scenario-based GPT-5.4 vs Opus 4.6 comparison |
| 144 | +- PR B deterministic suites still own auth/proxy/DNS and full-access truthfulness evidence |
| 145 | + |
| 146 | +## Goal-to-evidence map |
| 147 | + |
| 148 | +| Completion gate item | Primary owner | Review artifact | |
| 149 | +| ---------------------------------------- | ------------- | ------------------------------------------------------------------- | |
| 150 | +| No plan-only stalls | PR A | strict-agentic runtime tests and `approval-turn-tool-followthrough` | |
| 151 | +| No fake progress or fake tool completion | PR A + PR D | parity fake-success count plus scenario-level report details | |
| 152 | +| No false `/elevated full` guidance | PR B | deterministic runtime-truthfulness suites | |
| 153 | +| Replay/liveness failures remain explicit | PR C + PR D | lifecycle/replay suites plus `compaction-retry-mutating-tool` | |
| 154 | +| GPT-5.4 matches or beats Opus 4.6 | PR D | `qa-agentic-parity-report.md` and `qa-agentic-parity-summary.json` | |
| 155 | + |
| 156 | +## Reviewer shorthand: before vs after |
| 157 | + |
| 158 | +| User-visible problem before | Review signal after | |
| 159 | +| ----------------------------------------------------------- | --------------------------------------------------------------------------------------- | |
| 160 | +| GPT-5.4 stopped after planning | PR A shows act-or-block behavior instead of commentary-only completion | |
| 161 | +| Tool use felt brittle with strict OpenAI/Codex schemas | PR C keeps tool registration and parameter-free invocation predictable | |
| 162 | +| `/elevated full` hints were sometimes misleading | PR B ties guidance to actual runtime capability and blocked reasons | |
| 163 | +| Long tasks could disappear into replay/compaction ambiguity | PR C emits explicit paused, blocked, abandoned, and replay-invalid state | |
| 164 | +| Parity claims were anecdotal | PR D produces a report plus JSON verdict with the same scenario coverage on both models | |
0 commit comments