Skip to content

Commit 00b3935

Browse files
Kowserclaude
Kowser
and
claude
committed
Add per-PR agent-e2e workflow (release JAR + CLI + mcp-testkit)
Port of the Python SDK's implemented agent-e2e workflow. Kept verbatim: pull_request + workflow_dispatch triggers, per-ref concurrency with cancel-in-progress, pinned AGENTSPAN_VERSION 0.4.0 (server JAR from the GitHub release, cached by version; linux CLI binary), temurin 21, hard 90s health gate that dumps server.log on failure, executed>0 junit guard (defense-in-depth here — the TS suites fail hard when the server is down), junit + HTML + server.log artifacts at 14-day retention, 45-min timeout, job-level OPENAI_API_KEY/ANTHROPIC_API_KEY secrets (must be added to this repo's settings before the first live run). JS deltas: Node 22 + npm ci + npm run build + npm run test:agent-e2e as the test step; Python setup retained only for mcp-testkit; HTML report via npx tsx e2e/generate-report.ts; no CONDUCTOR_MP_START_METHOD (Python multiprocessing concern, no JS analog). Rehearsed locally: actionlint clean; health gate passed against the booted 0.4.0 JAR; guard correctly fails an all-skipped junit (0/196) and the report generator renders the real junit output. The test:agent-e2e script pins jest-junit's output to results/junit-e2e.xml via env vars — the package.json jest-junit block (reports/jest-junit.xml) outranks reporter options, and env vars outrank the block. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent b9de744 commit 00b3935

3 files changed

Lines changed: 151 additions & 1 deletion

File tree

‎.github/workflows/agent-e2e.yml‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
name: Agent E2E
2+
3+
# Runs the agent e2e suites (e2e/) against the released Agentspan
4+
# server JAR — a full Conductor server with the agent runtime baked in.
5+
# Port of the Python SDK's proven agent-e2e workflow; JS deltas only
6+
# (Node toolchain, jest runner; Python kept solely for mcp-testkit).
7+
#
8+
# These tests call real LLMs via the OPENAI_API_KEY / ANTHROPIC_API_KEY
9+
# repo secrets. Fork PRs cannot see repo secrets, so for them the run
10+
# fails at the silently-empty guard rather than passing vacuously.
11+
12+
on: [pull_request, workflow_dispatch]
13+
14+
concurrency:
15+
group: agent-e2e-${{ github.ref }}
16+
cancel-in-progress: true
17+
18+
env:
19+
AGENTSPAN_VERSION: "0.4.0" # pinned server/CLI release — bump deliberately
20+
21+
jobs:
22+
agent-e2e:
23+
runs-on: ubuntu-latest
24+
timeout-minutes: 45
25+
env:
26+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
27+
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
28+
AGENTSPAN_SERVER_URL: http://localhost:8080/api
29+
AGENTSPAN_CLI_PATH: ${{ github.workspace }}/agentspan
30+
steps:
31+
- name: Checkout code
32+
uses: actions/checkout@v4
33+
34+
- name: Set up Node
35+
uses: actions/setup-node@v4
36+
with:
37+
node-version: 22
38+
cache: npm
39+
40+
- name: Set up Python (mcp-testkit only)
41+
uses: actions/setup-python@v5
42+
with:
43+
python-version: '3.12'
44+
cache: 'pip'
45+
46+
- name: Set up Java
47+
uses: actions/setup-java@v4
48+
with:
49+
distribution: temurin
50+
java-version: '21'
51+
52+
- name: Cache server JAR
53+
id: jar_cache
54+
uses: actions/cache@v4
55+
with:
56+
path: agentspan-server.jar
57+
key: agentspan-server-${{ env.AGENTSPAN_VERSION }}
58+
59+
- name: Download server JAR from release
60+
if: steps.jar_cache.outputs.cache-hit != 'true'
61+
env:
62+
GH_TOKEN: ${{ github.token }}
63+
run: |
64+
gh release download "v${AGENTSPAN_VERSION}" --repo agentspan-ai/agentspan \
65+
--pattern "agentspan-server-${AGENTSPAN_VERSION}.jar" --output agentspan-server.jar
66+
67+
- name: Download CLI binary from release
68+
env:
69+
GH_TOKEN: ${{ github.token }}
70+
run: |
71+
gh release download "v${AGENTSPAN_VERSION}" --repo agentspan-ai/agentspan \
72+
--pattern "agentspan_linux_amd64" --output agentspan
73+
chmod +x agentspan
74+
75+
- name: Install SDK deps and build
76+
run: |
77+
npm ci
78+
npm run build
79+
80+
- name: Install mcp-testkit
81+
run: |
82+
python -m pip install --upgrade pip
83+
pip install mcp-testkit
84+
85+
- name: Start mcp-testkit
86+
run: |
87+
mcp-testkit --transport http --port 3001 &
88+
sleep 2
89+
90+
- name: Start server
91+
run: |
92+
java -jar agentspan-server.jar --server.port=8080 > server.log 2>&1 &
93+
94+
- name: Wait for server health
95+
run: |
96+
for i in $(seq 1 45); do
97+
if curl -sf http://localhost:8080/health | grep -q '"healthy"[[:space:]]*:[[:space:]]*true'; then
98+
echo "server healthy after ~$((i*2))s"; exit 0
99+
fi
100+
sleep 2
101+
done
102+
echo "::error::server failed to become healthy within 90s"
103+
tail -100 server.log
104+
exit 1
105+
106+
- name: Run e2e suites
107+
run: |
108+
mkdir -p results
109+
npm run test:agent-e2e
110+
111+
# The TS suites fail hard when the server is down (no session-skip),
112+
# so this guard is defense-in-depth against a future gate regression
113+
# or a secrets-less fork run skipping everything.
114+
- name: Guard against silently-empty runs
115+
if: always()
116+
run: |
117+
python - <<'EOF'
118+
import glob
119+
import sys
120+
import xml.etree.ElementTree as ET
121+
122+
total = executed = 0
123+
for path in glob.glob("results/junit-*.xml"):
124+
root = ET.parse(path).getroot()
125+
for suite in root.iter("testsuite"):
126+
t = int(suite.get("tests", 0))
127+
sk = int(suite.get("skipped", 0))
128+
total += t
129+
executed += t - sk
130+
print(f"executed {executed}/{total} tests")
131+
sys.exit(0 if executed > 0 else 1)
132+
EOF
133+
134+
- name: Generate HTML report
135+
if: always()
136+
run: |
137+
npx tsx e2e/generate-report.ts results/junit-e2e.xml results/report.html || true
138+
139+
- name: Upload results
140+
if: always()
141+
uses: actions/upload-artifact@v4
142+
with:
143+
name: agent-e2e-results
144+
path: |
145+
results/
146+
server.log
147+
retention-days: 14

‎jest.e2e.config.mjs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,9 @@ export default {
1313
// Upstream ran 3 vitest forks (credential names are unique per suite; 3
1414
// keeps server load manageable on the shared SQLite-backed Conductor).
1515
maxWorkers: 3,
16+
// The package.json "jest-junit" block outranks these reporter options, so
17+
// the test:agent-e2e script pins JEST_JUNIT_OUTPUT_DIR/NAME env vars
18+
// (which outrank everything) to results/junit-e2e.xml.
1619
reporters: [
1720
"default",
1821
["jest-junit", { outputDirectory: "results", outputName: "junit-e2e.xml" }],

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@
8080
"test:integration:v4": "cross-env ORKES_BACKEND_VERSION=4 npm run test:integration:base --",
8181
"test:integration:v5:batch": "cross-env ORKES_BACKEND_VERSION=5 node scripts/run-integration-batch.mjs",
8282
"test:integration:v4:batch": "cross-env ORKES_BACKEND_VERSION=4 node scripts/run-integration-batch.mjs",
83-
"test:agent-e2e": "jest --config jest.e2e.config.mjs --forceExit",
83+
"test:agent-e2e": "cross-env JEST_JUNIT_OUTPUT_DIR=results JEST_JUNIT_OUTPUT_NAME=junit-e2e.xml jest --config jest.e2e.config.mjs --forceExit",
8484
"ci": "npm run lint && npm run test",
8585
"build": "tsup && npm run verify:dist",
8686
"verify:dist": "node scripts/verify-dist.mjs",

0 commit comments

Comments
 (0)