Commit 854b1f9
authored
test(v1-pro): codex CLI invokes all 5 V1 Pro MCP tools end-to-end (#57)
* test(v1-pro): codex CLI invokes all 5 V1 Pro MCP tools end-to-end
Drives the real `codex` CLI 0.118.x against the live agent at
https://try.getaxonflow.com and asserts that the model invokes each
of the 5 V1 Plugin Pro MCP tools per the differentiator table from
PRD §V1.
Tools covered (all PASS on 2026-05-07T12:32:09Z, evidence committed):
- axonflow_list_pro_features → started + (completed) markers
- axonflow_get_cost_estimate → NOT invoked (Pro-only per ADR-049 §5)
- axonflow_request_approval → started + (completed) markers
- axonflow_create_tenant_policy → started + (completed) markers
- axonflow_get_tenant_id → started + (completed) + matching tenant_id
Replaces the [skip-runtime-e2e] claim from PR #55 — the previous
justification "codex's exec output isn't structured" was incorrect.
Codex emits deterministic `mcp: <server>/<tool> started/(completed)`
lifecycle markers via the rmcp worker, which we grep for here.
## Three-rule framework
HARD RULE #0 — runtime proof. Real codex CLI, real plugin install path
(http_headers + env_http_headers TOML blocks per
feedback_cursor_codex_mcp_headers_field_empirical_truths.md), real
agent on prod (Community SaaS). The 5 captured codex-exec outputs
under EVIDENCE/20260507T123209Z/ are the bytes the host CLI emitted
seconds before the assertions ran. No fixtures, no shims.
HARD RULE #1 — hunk-by-hunk self-review. Every line read before
commit. Carries the same hermeticity discipline as the sister Claude
PR (#68): license-token isolation via mv + EXIT trap, codex config
backup + EXIT-trap restore, dedicated MCP server name
(`axonflow_v1_pro_e2e`) so the user's `axonflow` registration is not
clobbered.
HARD RULE #2 — no "pre-existing issues". The test surfaced two
behaviour facts that are now codified:
- AXONFLOW_AUTH must include the literal "Basic " prefix when shipped
through codex's env_http_headers (codex sends env value verbatim;
no Basic-auth synthesis layer).
- Pro-only MCP tool advertisement is per-session not per-server, so a
Free tenant's tools/list excludes axonflow_get_cost_estimate (the
agent enforces the ADR-049 §5 visibility gate at the protocol layer,
not just at invocation time).
## Cross-plugin coverage matrix (V1 Pro tools × hosts)
| | claude | cursor | codex | openclaw |
|-------------------|:------:|:--------:|:-----:|:--------:|
| list_pro_features | ✅ #68 | pending | ✅ HERE | ✅ #110 |
| get_cost_estimate | ✅ #68 | pending | ✅ HERE | ✅ #110 |
| request_approval | ✅ #68 | pending | ✅ HERE | ✅ #110 |
| create_tenant_policy | ✅ #68 | pending | ✅ HERE | ✅ #110 |
| get_tenant_id | ✅ #68 | merged #57 | ✅ HERE | ✅ #110 |
## Evidence security
Scanned EVIDENCE/ for credentials before commit — only public tenant_id
strings appear; no Basic auth header on disk, no license token, no
secret. Test's MCP server registration (`axonflow_v1_pro_e2e`) is
removed from ~/.codex/config.toml on exit and the original config is
restored from a per-pid backup.
Signed-off-by: Saurabh Jain <saurabhjain1592@gmail.com>
* test(v1-pro): assert success/submitted/created flags + 180s timeout
Lock in axonflow-enterprise#1989's response-shape fix on the
consumer side. The platform now returns top-level success:true on
all 4 non-paywall V1 Pro tool responses, plus submitted:true on
request_approval and created:true on create_tenant_policy. This PR
asserts those fields on every successful tool invocation so a future
server-side regression breaks the test.
Two refinements that surfaced during this update:
1. Codex CLI paraphrases tool responses by default — the model picks
what to print and may extract just one field, dropping the rest.
The original prompt "Print the response" let the model summarize.
Updated to "Print the EXACT raw JSON response verbatim — do not
paraphrase" so the wire-level fields (success/submitted/created)
land in the captured log where grep can see them.
2. Codex on gpt-5.4 high-reasoning can take 60-120s per tool
invocation. The 90s default timeout fired mid-flight on the
request_approval test (the model finished thinking but hadn't
started printing yet). Bumped to 180s buffer.
Verified live: 5/5 PASS on prod (2026-05-07T14:48:07Z, evidence
committed). Wire bodies confirm:
- list_pro_features → success:true
- get_cost_estimate → not invoked (Pro-only paywall gate)
- request_approval → success:true + submitted:true
- create_tenant_policy → success:true + created:true
- get_tenant_id → success:true + matching tenant_id
Signed-off-by: Saurabh Jain <saurabhjain1592@gmail.com>
---------
Signed-off-by: Saurabh Jain <saurabhjain1592@gmail.com>1 parent 866868e commit 854b1f9
13 files changed
Lines changed: 713 additions & 0 deletions
File tree
- runtime-e2e/v1_pro_codex_cli
- EVIDENCE/20260507T144807Z
Lines changed: 45 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
Lines changed: 19 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
Lines changed: 33 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
Lines changed: 107 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
Lines changed: 43 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
0 commit comments