Skip to content

Commit fe44caf

Browse files
Trecekclaude
Trecek
and
claude
authored
Implementation Plan: T5-P3-A3-WP3 Produce Merge-Blocking CI Tests for Codex Deterministic Conformance (#4149)
## Summary Add `tests/execution/backends/test_codex_deterministic_conformance.py` with four test classes that seal the `CodexEventType`/`CodexItemType` vocabulary, hook event formats, and config.toml schema template against undetected drift. Register `--update-fixtures` in the existing `pytest_addoption()` in `tests/conftest.py` and expose it as a pytest fixture. All tests are pure Python assertions against committed JSON fixtures — no live CLI, no subprocess, no network I/O. Closes #4011 ## Implementation Plan Plan file: `/home/talon/projects/autoskillit-runs/impl-20260628-181308-999735/.autoskillit/temp/make-plan/t5_p3_a3_wp3_deterministic_conformance_plan_2026-06-28_181500.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) via AutoSkillit <!-- autoskillit:pipeline-signature steps=prepare_pr,run_arch_lenses,compose_pr,annotate_pr_diff,review_pr --> ## Token Usage Summary | Step | Model | count | uncached | output | cache_read | peak_ctx | turns | cache_write | time | |------|-------|-------|----------|--------|------------|----------|-------|-------------|------| | plan* | opus[1m] | 1 | 56 | 16.0k | 1.4M | 103.4k | 53 | 85.0k | 12m 28s | | verify* | sonnet | 1 | 92 | 12.2k | 513.0k | 58.1k | 28 | 39.6k | 6m 32s | | implement* | MiniMax-M3 | 1 | 70.2k | 8.8k | 1.3M | 0 | 62 | 0 | 3m 19s | | fix* | sonnet | 1 | 150 | 8.5k | 898.8k | 66.6k | 45 | 48.0k | 8m 21s | | audit_impl* | sonnet | 1 | 52 | 18.9k | 221.1k | 52.8k | 16 | 54.6k | 8m 27s | | prepare_pr* | MiniMax-M3 | 1 | 45.2k | 2.6k | 202.4k | 0 | 14 | 0 | 43s | | compose_pr* | MiniMax-M3 | 1 | 36.5k | 1.6k | 104.3k | 0 | 10 | 0 | 33s | | **Total** | | | 152.3k | 68.7k | 4.7M | 103.4k | | 227.2k | 40m 25s | \* *Step used a non-Anthropic provider; caching behavior may differ.* ## Token Efficiency | Step | LoC Changed | cache_read/LoC | cache_write/LoC | output/LoC | |------|-------------|----------------|-----------------|------------| | plan | 0 | — | — | — | | verify | 0 | — | — | — | | implement | 259 | 5210.4 | 0.0 | 34.1 | | fix | 140 | 6420.3 | 343.2 | 60.9 | | audit_impl | 0 | — | — | — | | prepare_pr | 0 | — | — | — | | compose_pr | 0 | — | — | — | | **Total** | **399** | 11787.1 | 569.5 | 172.3 | ## Model Usage Breakdown | Model | steps | uncached | output | cache_read | cache_write | time | |-------|-------|----------|--------|------------|-------------|------| | opus[1m] | 1 | 56 | 16.0k | 1.4M | 85.0k | 12m 28s | | sonnet | 3 | 294 | 39.6k | 1.6M | 142.3k | 23m 22s | | MiniMax-M3 | 3 | 151.9k | 13.1k | 1.7M | 0 | 4m 35s | --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent f55656b commit fe44caf

11 files changed

Lines changed: 404 additions & 0 deletions

‎tests/conftest.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -468,6 +468,12 @@ def pytest_addoption(parser: pytest.Parser) -> None:
468468
default=None,
469469
help="Git base ref for changed-file detection (overrides AUTOSKILLIT_TEST_BASE_REF).",
470470
)
471+
parser.addoption(
472+
"--update-fixtures",
473+
action="store_true",
474+
default=False,
475+
help="Regenerate deterministic conformance fixtures in-place instead of asserting.",
476+
)
471477

472478

473479
def pytest_configure(config: pytest.Config) -> None:

‎tests/execution/AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -153,5 +153,6 @@ Subprocess integration, headless session, process lifecycle, and session result
153153
| `test_codex_stream_parser.py` | Full test suite for CodexStreamParser: happy-path, item parsing, degradation, fixture-driven integration, protocol conformance |
154154
| `test_codex_result_parser.py` | Tests for CodexResultParser |
155155
| `test_codex_config.py` | Tests for TOML read/write primitives, _is_autoskillit_registered, and ensure_codex_mcp_registered |
156+
| `test_codex_deterministic_conformance.py` | Sealed-enum vocabulary, hook event format, and config.toml schema template conformance tests with --update-fixtures review gate |
156157
| `test_cmd_builder.py` | CmdBuilder ordering invariant and CmdSpec origin tests |
157158
| `test_coding_agent_backend_conformance.py` | Parametrized conformance tests for all CodingAgentBackend implementations via BackendContractBase |

‎tests/execution/backends/conftest.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,17 @@
1313
import re
1414
from pathlib import Path
1515

16+
import pytest
17+
1618
from autoskillit.hook_registry import HOOK_REGISTRY
1719

20+
21+
@pytest.fixture
22+
def update_fixtures(request: pytest.FixtureRequest) -> bool:
23+
"""Expose --update-fixtures CLI flag as a fixture."""
24+
return bool(request.config.getoption("--update-fixtures"))
25+
26+
1827
# Local mirror of _SKIP_CODEX_STATUSES from autoskillit.execution.backends._codex_hooks.
1928
# Defined here so the conftest has no import dependency on execution.backends (which
2029
# would create a tests/ -> execution/ IL cycle for the backends subdirectory).

‎tests/execution/backends/fixtures/codex_ndjson/event_item.completed.json‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
{
2+
"$schema": "http://json-schema.org/draft-07/schema#",
3+
"additionalProperties": false,
4+
"properties": {
5+
"item": {
6+
"properties": {
7+
"type": {
8+
"type": "string"
9+
}
10+
},
11+
"required": [
12+
"type"
13+
],
14+
"type": "object"
15+
},
16+
"type": {
17+
"const": "item.completed",
18+
"type": "string"
19+
}
20+
},
21+
"required": [
22+
"type",
23+
"item"
24+
],
25+
"type": "object"
26+
}

‎tests/execution/backends/fixtures/codex_ndjson/event_item.started.json‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"$schema": "http://json-schema.org/draft-07/schema#",
3+
"additionalProperties": false,
4+
"properties": {
5+
"type": {
6+
"const": "item.started",
7+
"type": "string"
8+
}
9+
},
10+
"required": [
11+
"type"
12+
],
13+
"type": "object"
14+
}

‎tests/execution/backends/fixtures/codex_ndjson/event_item.updated.json‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"$schema": "http://json-schema.org/draft-07/schema#",
3+
"additionalProperties": false,
4+
"properties": {
5+
"type": {
6+
"const": "item.updated",
7+
"type": "string"
8+
}
9+
},
10+
"required": [
11+
"type"
12+
],
13+
"type": "object"
14+
}

‎tests/execution/backends/fixtures/codex_ndjson/event_thread.started.json‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{
2+
"$schema": "http://json-schema.org/draft-07/schema#",
3+
"additionalProperties": false,
4+
"properties": {
5+
"thread_id": {
6+
"type": "string"
7+
},
8+
"type": {
9+
"const": "thread.started",
10+
"type": "string"
11+
}
12+
},
13+
"required": [
14+
"type",
15+
"thread_id"
16+
],
17+
"type": "object"
18+
}

‎tests/execution/backends/fixtures/codex_ndjson/event_turn.completed.json‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{
2+
"$schema": "http://json-schema.org/draft-07/schema#",
3+
"additionalProperties": false,
4+
"properties": {
5+
"type": {
6+
"const": "turn.completed",
7+
"type": "string"
8+
},
9+
"usage": {
10+
"type": "object"
11+
}
12+
},
13+
"required": [
14+
"type",
15+
"usage"
16+
],
17+
"type": "object"
18+
}

‎tests/execution/backends/fixtures/codex_ndjson/event_turn.failed.json‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
{
2+
"$schema": "http://json-schema.org/draft-07/schema#",
3+
"additionalProperties": false,
4+
"properties": {
5+
"error": {
6+
"oneOf": [
7+
{
8+
"properties": {
9+
"code": {
10+
"type": "string"
11+
},
12+
"message": {
13+
"type": "string"
14+
}
15+
},
16+
"required": [
17+
"message"
18+
],
19+
"type": "object"
20+
},
21+
{
22+
"type": "string"
23+
}
24+
]
25+
},
26+
"type": {
27+
"const": "turn.failed",
28+
"type": "string"
29+
}
30+
},
31+
"required": [
32+
"type",
33+
"error"
34+
],
35+
"type": "object"
36+
}

‎tests/execution/backends/fixtures/codex_ndjson/event_turn.started.json‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"$schema": "http://json-schema.org/draft-07/schema#",
3+
"additionalProperties": false,
4+
"properties": {
5+
"type": {
6+
"const": "turn.started",
7+
"type": "string"
8+
}
9+
},
10+
"required": [
11+
"type"
12+
],
13+
"type": "object"
14+
}

0 commit comments

Comments
 (0)