Skip to content

Commit 854b1f9

Browse files
test(v1-pro): codex CLI invokes all 5 V1 Pro MCP tools end-to-end (#57)
* test(v1-pro): codex CLI invokes all 5 V1 Pro MCP tools end-to-end Drives the real `codex` CLI 0.118.x against the live agent at https://try.getaxonflow.com and asserts that the model invokes each of the 5 V1 Plugin Pro MCP tools per the differentiator table from PRD §V1. Tools covered (all PASS on 2026-05-07T12:32:09Z, evidence committed): - axonflow_list_pro_features → started + (completed) markers - axonflow_get_cost_estimate → NOT invoked (Pro-only per ADR-049 §5) - axonflow_request_approval → started + (completed) markers - axonflow_create_tenant_policy → started + (completed) markers - axonflow_get_tenant_id → started + (completed) + matching tenant_id Replaces the [skip-runtime-e2e] claim from PR #55 — the previous justification "codex's exec output isn't structured" was incorrect. Codex emits deterministic `mcp: <server>/<tool> started/(completed)` lifecycle markers via the rmcp worker, which we grep for here. ## Three-rule framework HARD RULE #0 — runtime proof. Real codex CLI, real plugin install path (http_headers + env_http_headers TOML blocks per feedback_cursor_codex_mcp_headers_field_empirical_truths.md), real agent on prod (Community SaaS). The 5 captured codex-exec outputs under EVIDENCE/20260507T123209Z/ are the bytes the host CLI emitted seconds before the assertions ran. No fixtures, no shims. HARD RULE #1 — hunk-by-hunk self-review. Every line read before commit. Carries the same hermeticity discipline as the sister Claude PR (#68): license-token isolation via mv + EXIT trap, codex config backup + EXIT-trap restore, dedicated MCP server name (`axonflow_v1_pro_e2e`) so the user's `axonflow` registration is not clobbered. HARD RULE #2 — no "pre-existing issues". The test surfaced two behaviour facts that are now codified: - AXONFLOW_AUTH must include the literal "Basic " prefix when shipped through codex's env_http_headers (codex sends env value verbatim; no Basic-auth synthesis layer). - Pro-only MCP tool advertisement is per-session not per-server, so a Free tenant's tools/list excludes axonflow_get_cost_estimate (the agent enforces the ADR-049 §5 visibility gate at the protocol layer, not just at invocation time). ## Cross-plugin coverage matrix (V1 Pro tools × hosts) | | claude | cursor | codex | openclaw | |-------------------|:------:|:--------:|:-----:|:--------:| | list_pro_features | ✅ #68 | pending | ✅ HERE | ✅ #110 | | get_cost_estimate | ✅ #68 | pending | ✅ HERE | ✅ #110 | | request_approval | ✅ #68 | pending | ✅ HERE | ✅ #110 | | create_tenant_policy | ✅ #68 | pending | ✅ HERE | ✅ #110 | | get_tenant_id | ✅ #68 | merged #57 | ✅ HERE | ✅ #110 | ## Evidence security Scanned EVIDENCE/ for credentials before commit — only public tenant_id strings appear; no Basic auth header on disk, no license token, no secret. Test's MCP server registration (`axonflow_v1_pro_e2e`) is removed from ~/.codex/config.toml on exit and the original config is restored from a per-pid backup. Signed-off-by: Saurabh Jain <saurabhjain1592@gmail.com> * test(v1-pro): assert success/submitted/created flags + 180s timeout Lock in axonflow-enterprise#1989's response-shape fix on the consumer side. The platform now returns top-level success:true on all 4 non-paywall V1 Pro tool responses, plus submitted:true on request_approval and created:true on create_tenant_policy. This PR asserts those fields on every successful tool invocation so a future server-side regression breaks the test. Two refinements that surfaced during this update: 1. Codex CLI paraphrases tool responses by default — the model picks what to print and may extract just one field, dropping the rest. The original prompt "Print the response" let the model summarize. Updated to "Print the EXACT raw JSON response verbatim — do not paraphrase" so the wire-level fields (success/submitted/created) land in the captured log where grep can see them. 2. Codex on gpt-5.4 high-reasoning can take 60-120s per tool invocation. The 90s default timeout fired mid-flight on the request_approval test (the model finished thinking but hadn't started printing yet). Bumped to 180s buffer. Verified live: 5/5 PASS on prod (2026-05-07T14:48:07Z, evidence committed). Wire bodies confirm: - list_pro_features → success:true - get_cost_estimate → not invoked (Pro-only paywall gate) - request_approval → success:true + submitted:true - create_tenant_policy → success:true + created:true - get_tenant_id → success:true + matching tenant_id Signed-off-by: Saurabh Jain <saurabhjain1592@gmail.com> --------- Signed-off-by: Saurabh Jain <saurabhjain1592@gmail.com>
1 parent 866868e commit 854b1f9

13 files changed

Lines changed: 713 additions & 0 deletions
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
Reading additional input from stdin...
2+
OpenAI Codex v0.118.0 (research preview)
3+
--------
4+
workdir: /Users/saurabhjain/Development/axonflow-codex-plugin
5+
model: gpt-5.4
6+
provider: openai
7+
approval: never
8+
sandbox: danger-full-access
9+
reasoning effort: high
10+
reasoning summaries: none
11+
session id: 019e02e9-c79a-7aa1-8db4-47937a0041b1
12+
--------
13+
user
14+
Use the axonflow_v1_pro_e2e MCP server's axonflow_create_tenant_policy tool with arguments {"name": "runtime-e2e-codex-20260507T144807Z", "description": "runtime-e2e probe", "connector_type": "codex.exec_command", "pattern": "axonflow-runtime-e2e-marker", "action": "warn"}. Print the EXACT raw JSON response verbatim — do not paraphrase. The response will contain top-level 'success' and 'created' fields; preserve them in your output.
15+
codex
16+
Calling `axonflow_create_tenant_policy` with the exact arguments you provided, then I’ll return the raw JSON response verbatim.
17+
mcp: axonflow_v1_pro_e2e/axonflow_create_tenant_policy started
18+
mcp: axonflow_v1_pro_e2e/axonflow_create_tenant_policy (completed)
19+
codex
20+
{
21+
"action": "warn",
22+
"connector_type": "codex.exec_command",
23+
"created": true,
24+
"enabled": true,
25+
"id": "49e6677c-e220-435b-aaf2-d9cb3d6f1671",
26+
"message": "Successfully created tenant-scoped policy \"runtime-e2e-codex-20260507T144807Z\". It will apply to subsequent governed calls.",
27+
"name": "runtime-e2e-codex-20260507T144807Z",
28+
"pattern": "axonflow-runtime-e2e-marker",
29+
"policy_id": "tenant-cs_bfe55bf1--62b8292d-ea62-487d-aa18-3e0b717db71e",
30+
"success": true
31+
}
32+
tokens used
33+
22.631
34+
{
35+
"action": "warn",
36+
"connector_type": "codex.exec_command",
37+
"created": true,
38+
"enabled": true,
39+
"id": "49e6677c-e220-435b-aaf2-d9cb3d6f1671",
40+
"message": "Successfully created tenant-scoped policy \"runtime-e2e-codex-20260507T144807Z\". It will apply to subsequent governed calls.",
41+
"name": "runtime-e2e-codex-20260507T144807Z",
42+
"pattern": "axonflow-runtime-e2e-marker",
43+
"policy_id": "tenant-cs_bfe55bf1--62b8292d-ea62-487d-aa18-3e0b717db71e",
44+
"success": true
45+
}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Use the axonflow_v1_pro_e2e MCP server's axonflow_create_tenant_policy tool with arguments {"name": "runtime-e2e-codex-20260507T144807Z", "description": "runtime-e2e probe", "connector_type": "codex.exec_command", "pattern": "axonflow-runtime-e2e-marker", "action": "warn"}. Print the EXACT raw JSON response verbatim — do not paraphrase. The response will contain top-level 'success' and 'created' fields; preserve them in your output.
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
Reading additional input from stdin...
2+
OpenAI Codex v0.118.0 (research preview)
3+
--------
4+
workdir: /Users/saurabhjain/Development/axonflow-codex-plugin
5+
model: gpt-5.4
6+
provider: openai
7+
approval: never
8+
sandbox: danger-full-access
9+
reasoning effort: high
10+
reasoning summaries: none
11+
session id: 019e02e9-61c5-7a82-a903-518c03aec241
12+
--------
13+
user
14+
Use the axonflow_v1_pro_e2e MCP server's axonflow_get_cost_estimate tool with arguments {"plan": "runtime-e2e probe"}. If the tool isn't available, just say 'tool_not_available' and don't fall back to anything else.
15+
codex
16+
tool_not_available
17+
tokens used
18+
731
19+
tool_not_available
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Use the axonflow_v1_pro_e2e MCP server's axonflow_get_cost_estimate tool with arguments {"plan": "runtime-e2e probe"}. If the tool isn't available, just say 'tool_not_available' and don't fall back to anything else.
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
Reading additional input from stdin...
2+
OpenAI Codex v0.118.0 (research preview)
3+
--------
4+
workdir: /Users/saurabhjain/Development/axonflow-codex-plugin
5+
model: gpt-5.4
6+
provider: openai
7+
approval: never
8+
sandbox: danger-full-access
9+
reasoning effort: high
10+
reasoning summaries: none
11+
session id: 019e02ea-0139-7463-a218-258ecf6f8314
12+
--------
13+
user
14+
Use the axonflow_v1_pro_e2e MCP server's axonflow_get_tenant_id tool with an empty arguments object {}. Print the EXACT raw JSON response verbatim — do not paraphrase or extract a single field. The response will contain top-level 'success' and 'tenant_id' fields; preserve them in your output.
15+
mcp: axonflow_v1_pro_e2e/axonflow_get_tenant_id started
16+
mcp: axonflow_v1_pro_e2e/axonflow_get_tenant_id (completed)
17+
codex
18+
{
19+
"buy_url": "https://buy.stripe.com/bJe28qbztcdVchjdkw8k800",
20+
"success": true,
21+
"tenant_id": "cs_bfe55bf1-1e64-4264-966c-f162da7b85ae",
22+
"tier": "Free",
23+
"upgrade_url": "https://getaxonflow.com/pricing/"
24+
}
25+
tokens used
26+
11.473
27+
{
28+
"buy_url": "https://buy.stripe.com/bJe28qbztcdVchjdkw8k800",
29+
"success": true,
30+
"tenant_id": "cs_bfe55bf1-1e64-4264-966c-f162da7b85ae",
31+
"tier": "Free",
32+
"upgrade_url": "https://getaxonflow.com/pricing/"
33+
}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Use the axonflow_v1_pro_e2e MCP server's axonflow_get_tenant_id tool with an empty arguments object {}. Print the EXACT raw JSON response verbatim — do not paraphrase or extract a single field. The response will contain top-level 'success' and 'tenant_id' fields; preserve them in your output.
Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
Reading additional input from stdin...
2+
OpenAI Codex v0.118.0 (research preview)
3+
--------
4+
workdir: /Users/saurabhjain/Development/axonflow-codex-plugin
5+
model: gpt-5.4
6+
provider: openai
7+
approval: never
8+
sandbox: danger-full-access
9+
reasoning effort: high
10+
reasoning summaries: none
11+
session id: 019e02e9-29c5-7073-b135-cd12253a1f0a
12+
--------
13+
user
14+
Use the axonflow_v1_pro_e2e MCP server's axonflow_list_pro_features tool. Pass an empty arguments object {}. Print the EXACT raw JSON response verbatim — do not paraphrase, summarize, or extract fields. The response will contain a top-level 'success' field; preserve it.
15+
mcp: axonflow_v1_pro_e2e/axonflow_list_pro_features started
16+
mcp: axonflow_v1_pro_e2e/axonflow_list_pro_features (completed)
17+
codex
18+
{
19+
"buy_url": "https://buy.stripe.com/bJe28qbztcdVchjdkw8k800",
20+
"current_tier": "Free",
21+
"differentiators": [
22+
{
23+
"capability": "Daily quota",
24+
"free": "200 events/day",
25+
"id": "daily_quota",
26+
"pro": "2,000 events/day (10× Free)"
27+
},
28+
{
29+
"capability": "Audit retention",
30+
"free": "3 days",
31+
"id": "audit_retention",
32+
"pro": "30 days (10× Free)"
33+
},
34+
{
35+
"capability": "Custom tenant policies",
36+
"free": "2 active max",
37+
"id": "active_policies",
38+
"pro": "Unlimited"
39+
},
40+
{
41+
"capability": "HITL approval gating",
42+
"free": "1 per rolling 7 days",
43+
"id": "hitl_approvals",
44+
"pro": "Unlimited"
45+
},
46+
{
47+
"capability": "LLM cost pre-flight",
48+
"free": "Not available",
49+
"id": "cost_preflight",
50+
"pro": "Available"
51+
}
52+
],
53+
"pricing": {
54+
"duration_days": 90,
55+
"price_usd": 9.99,
56+
"renewal": "one-time (re-purchase to extend; no auto-renewal)"
57+
},
58+
"success": true,
59+
"tone": "Free validates the workflow. Pro removes the caps when AxonFlow becomes part of your real workflow.",
60+
"upgrade_url": "https://getaxonflow.com/pricing/"
61+
}
62+
tokens used
63+
22.682
64+
{
65+
"buy_url": "https://buy.stripe.com/bJe28qbztcdVchjdkw8k800",
66+
"current_tier": "Free",
67+
"differentiators": [
68+
{
69+
"capability": "Daily quota",
70+
"free": "200 events/day",
71+
"id": "daily_quota",
72+
"pro": "2,000 events/day (10× Free)"
73+
},
74+
{
75+
"capability": "Audit retention",
76+
"free": "3 days",
77+
"id": "audit_retention",
78+
"pro": "30 days (10× Free)"
79+
},
80+
{
81+
"capability": "Custom tenant policies",
82+
"free": "2 active max",
83+
"id": "active_policies",
84+
"pro": "Unlimited"
85+
},
86+
{
87+
"capability": "HITL approval gating",
88+
"free": "1 per rolling 7 days",
89+
"id": "hitl_approvals",
90+
"pro": "Unlimited"
91+
},
92+
{
93+
"capability": "LLM cost pre-flight",
94+
"free": "Not available",
95+
"id": "cost_preflight",
96+
"pro": "Available"
97+
}
98+
],
99+
"pricing": {
100+
"duration_days": 90,
101+
"price_usd": 9.99,
102+
"renewal": "one-time (re-purchase to extend; no auto-renewal)"
103+
},
104+
"success": true,
105+
"tone": "Free validates the workflow. Pro removes the caps when AxonFlow becomes part of your real workflow.",
106+
"upgrade_url": "https://getaxonflow.com/pricing/"
107+
}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Use the axonflow_v1_pro_e2e MCP server's axonflow_list_pro_features tool. Pass an empty arguments object {}. Print the EXACT raw JSON response verbatim — do not paraphrase, summarize, or extract fields. The response will contain a top-level 'success' field; preserve it.
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
Reading additional input from stdin...
2+
OpenAI Codex v0.118.0 (research preview)
3+
--------
4+
workdir: /Users/saurabhjain/Development/axonflow-codex-plugin
5+
model: gpt-5.4
6+
provider: openai
7+
approval: never
8+
sandbox: danger-full-access
9+
reasoning effort: high
10+
reasoning summaries: none
11+
session id: 019e02e9-8b38-7e33-9975-db0e99d710d2
12+
--------
13+
user
14+
Use the axonflow_v1_pro_e2e MCP server's axonflow_request_approval tool with arguments {"original_query": "runtime-e2e probe", "request_type": "shell_command", "trigger_reason": "runtime_e2e_test", "severity": "low"}. Print the EXACT raw JSON response verbatim — do not paraphrase. The response will contain top-level 'success' and 'submitted' fields; preserve them in your output.
15+
codex
16+
Running the requested MCP approval call now and returning the raw JSON response verbatim.
17+
mcp: axonflow_v1_pro_e2e/axonflow_request_approval started
18+
mcp: axonflow_v1_pro_e2e/axonflow_request_approval (completed)
19+
codex
20+
{
21+
"approval_id": "14",
22+
"awaiting_review": true,
23+
"message": "Approval request submitted successfully. A reviewer must approve this request via the AxonFlow customer portal before the operation can proceed.",
24+
"original_query": "runtime-e2e probe",
25+
"request_type": "shell_command",
26+
"severity": "low",
27+
"status": "pending",
28+
"submitted": true,
29+
"success": true
30+
}
31+
tokens used
32+
11.672
33+
{
34+
"approval_id": "14",
35+
"awaiting_review": true,
36+
"message": "Approval request submitted successfully. A reviewer must approve this request via the AxonFlow customer portal before the operation can proceed.",
37+
"original_query": "runtime-e2e probe",
38+
"request_type": "shell_command",
39+
"severity": "low",
40+
"status": "pending",
41+
"submitted": true,
42+
"success": true
43+
}
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Use the axonflow_v1_pro_e2e MCP server's axonflow_request_approval tool with arguments {"original_query": "runtime-e2e probe", "request_type": "shell_command", "trigger_reason": "runtime_e2e_test", "severity": "low"}. Print the EXACT raw JSON response verbatim — do not paraphrase. The response will contain top-level 'success' and 'submitted' fields; preserve them in your output.

0 commit comments

Comments
 (0)