Skip to content

Commit 8ac2cdc

Browse files
Kowserclaude
Kowser
and
claude
committed
Add nightly agent e2e/integration CI against the released server JAR
New agent-e2e.yml workflow restores automated execution to the two agent test layers that had none after the merge: e2e/ (upstream ran it on every PR in the Agentspan monorepo CI; the job didn't survive the migration) and tests/integration/ai (never had CI anywhere). Instead of Gradle-building the server and Go-building the CLI like upstream, it downloads both prebuilt from the agentspan v0.4.0 GitHub release (pinned via AGENTSPAN_VERSION; endpoint surface verified identical to the SDK's source commit), boots the JAR on :8080, starts mcp-testkit for the MCP/HTTP tool suites, and runs both layers with junit output and the migrated HTML report generator. Nightly + workflow_dispatch, not per-PR: the suites call real LLMs (OPENAI_API_KEY / ANTHROPIC_API_KEY repo secrets, unavailable to fork PRs) and take ~25-40 min. Two guards prevent a green-but-empty run: a hard health gate on the server boot, and a post-run junit check that executed - skipped > 0 (e2e/conftest.py skips the whole session when the server is down). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent fc97b2a commit 8ac2cdc

1 file changed

Lines changed: 144 additions & 0 deletions

File tree

‎.github/workflows/agent-e2e.yml‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
name: Agent E2E
2+
3+
# Runs the agent e2e suites (e2e/) and agent integration tests
4+
# (tests/integration/ai) against the released Agentspan server JAR —
5+
# a full Conductor server with the agent runtime baked in.
6+
#
7+
# Not a per-PR gate: these tests call real LLMs (OPENAI_API_KEY /
8+
# ANTHROPIC_API_KEY repo secrets, which fork PRs cannot see) and take
9+
# ~25-40 minutes. Nightly + manual dispatch instead.
10+
11+
on:
12+
schedule:
13+
- cron: "0 6 * * *"
14+
workflow_dispatch: {}
15+
16+
concurrency:
17+
group: agent-e2e-${{ github.ref }}
18+
cancel-in-progress: true
19+
20+
env:
21+
AGENTSPAN_VERSION: "0.4.0" # pinned server/CLI release — bump deliberately
22+
23+
jobs:
24+
agent-e2e:
25+
runs-on: ubuntu-latest
26+
timeout-minutes: 45
27+
env:
28+
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
29+
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
30+
AGENTSPAN_SERVER_URL: http://localhost:8080/api
31+
AGENTSPAN_CLI_PATH: ${{ github.workspace }}/agentspan
32+
steps:
33+
- name: Checkout code
34+
uses: actions/checkout@v4
35+
36+
- name: Set up Python
37+
uses: actions/setup-python@v5
38+
with:
39+
python-version: '3.12'
40+
cache: 'pip'
41+
42+
- name: Set up Java
43+
uses: actions/setup-java@v4
44+
with:
45+
distribution: temurin
46+
java-version: '21'
47+
48+
- name: Cache server JAR
49+
id: jar_cache
50+
uses: actions/cache@v4
51+
with:
52+
path: agentspan-server.jar
53+
key: agentspan-server-${{ env.AGENTSPAN_VERSION }}
54+
55+
- name: Download server JAR from release
56+
if: steps.jar_cache.outputs.cache-hit != 'true'
57+
env:
58+
GH_TOKEN: ${{ github.token }}
59+
run: |
60+
gh release download "v${AGENTSPAN_VERSION}" --repo agentspan-ai/agentspan \
61+
--pattern "agentspan-server-${AGENTSPAN_VERSION}.jar" --output agentspan-server.jar
62+
63+
- name: Download CLI binary from release
64+
env:
65+
GH_TOKEN: ${{ github.token }}
66+
run: |
67+
gh release download "v${AGENTSPAN_VERSION}" --repo agentspan-ai/agentspan \
68+
--pattern "agentspan_linux_amd64" --output agentspan
69+
chmod +x agentspan
70+
71+
- name: Install SDK + test deps
72+
run: |
73+
python -m pip install --upgrade pip
74+
pip install -e '.[agents]'
75+
pip install pytest pytest-asyncio pytest-xdist pytest-rerunfailures mcp-testkit
76+
77+
- name: Start mcp-testkit
78+
run: |
79+
mcp-testkit --transport http --port 3001 &
80+
sleep 2
81+
82+
- name: Start server
83+
run: |
84+
java -jar agentspan-server.jar --server.port=8080 > server.log 2>&1 &
85+
86+
- name: Wait for server health
87+
run: |
88+
for i in $(seq 1 45); do
89+
if curl -sf http://localhost:8080/health | grep -q '"healthy"[[:space:]]*:[[:space:]]*true'; then
90+
echo "server healthy after ~$((i*2))s"; exit 0
91+
fi
92+
sleep 2
93+
done
94+
echo "::error::server failed to become healthy within 90s"
95+
tail -100 server.log
96+
exit 1
97+
98+
- name: Run agent integration tests
99+
run: |
100+
mkdir -p results
101+
pytest tests/integration/ai -v --junitxml=results/junit-integration.xml
102+
103+
- name: Run e2e suites
104+
run: |
105+
pytest e2e/ -v --tb=short -n 3 --dist=loadgroup \
106+
--junitxml=results/junit-e2e.xml
107+
108+
# e2e/conftest.py pytest.skip()s the whole session when the server is
109+
# unreachable — without this guard a boot failure after the health gate
110+
# (or a future gate regression) would yield a green job that ran nothing.
111+
- name: Guard against silently-empty runs
112+
if: always()
113+
run: |
114+
python - <<'EOF'
115+
import glob
116+
import sys
117+
import xml.etree.ElementTree as ET
118+
119+
total = executed = 0
120+
for path in glob.glob("results/junit-*.xml"):
121+
root = ET.parse(path).getroot()
122+
for suite in root.iter("testsuite"):
123+
t = int(suite.get("tests", 0))
124+
sk = int(suite.get("skipped", 0))
125+
total += t
126+
executed += t - sk
127+
print(f"executed {executed}/{total} tests")
128+
sys.exit(0 if executed > 0 else 1)
129+
EOF
130+
131+
- name: Generate HTML report
132+
if: always()
133+
run: |
134+
python e2e/report_generator.py results/junit-e2e.xml results/report.html || true
135+
136+
- name: Upload results
137+
if: always()
138+
uses: actions/upload-artifact@v4
139+
with:
140+
name: agent-e2e-results
141+
path: |
142+
results/
143+
server.log
144+
retention-days: 14

0 commit comments

Comments
 (0)