Skip to content

Commit 5ad1961

Browse files
committed
ci: Fix integration test timeouts with bounded nextest limit and job timeout
The integration CI jobs were hitting GitHub's 6-hour job limit, which caused silent cancellation rather than a real test failure. Two issues combined to cause this: 1. nextest's integration profile had terminate-after = 60, meaning a single hung test could run for 1200 × 60 = 72 000 s (20 h) before nextest force-killed it, far exceeding the 6-hour GHA limit. 2. The integration job had no explicit timeout-minutes, so a stalled job wasn't surfaced as a clear timeout failure. Fix both: drop terminate-after to 1 (each test gets exactly 20 min before nextest terminates it, which is generous for VM boot + test execution), and add timeout-minutes: 90 to the integration job so any build or runner hang fails cleanly rather than silently burning 6 hours. Assisted-by: OpenCode (Claude Sonnet 4.6) Signed-off-by: Colin Walters <walters@verbum.org>
1 parent c8fa590 commit 5ad1961

2 files changed

Lines changed: 18 additions & 2 deletions

File tree

.config/nextest.toml

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,12 @@ slow-timeout = { period = "300s", terminate-after = 2 }
2222
# Profile for integration tests — run with limited parallelism due to QEMU/KVM resources
2323
[profile.integration]
2424
test-threads = 2
25-
# 20 minutes for VM tests on slow CI runners
26-
slow-timeout = { period = "1200s", terminate-after = 60 }
25+
# 20 minutes per test; kill after 1 period to surface hangs quickly.
26+
# terminate-after = 1 means a test gets exactly 20 min before nextest force-kills it,
27+
# which is generous for VM boot + test execution on the slowest GHA runners.
28+
# With terminate-after = 60 (the old value) a single hung test could consume
29+
# 1200 × 60 = 72000 s (20 h), burning through the 6 h GHA job limit silently.
30+
slow-timeout = { period = "1200s", terminate-after = 1 }
2731
fail-fast = false
2832
failure-output = "immediate"
2933
success-output = "never"
@@ -44,3 +48,10 @@ store-failure-output = true
4448
[[profile.integration.overrides]]
4549
filter = 'test(/^privileged_/)'
4650
threads-required = 2
51+
52+
# privileged_fast_suite consolidates multiple sub-tests into a single VM boot,
53+
# so it needs a longer timeout than the default 20-minute single-test budget.
54+
# 45 minutes covers one VM boot + 10 sequential sub-tests on the slowest TCG runner.
55+
[[profile.integration.overrides]]
56+
filter = 'test(/^privileged_fast_suite/)'
57+
slow-timeout = { period = "2700s", terminate-after = 1 }

.github/workflows/ci.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,11 @@ jobs:
9393
name: Integration tests (${{ matrix.name }})
9494
needs: smoke
9595
runs-on: ubuntu-24.04
96+
# Generous ceiling: nextest kills individual tests at 20 min
97+
# (terminate-after = 1 × period = 1200s in .config/nextest.toml), so
98+
# this job-level limit only fires if the build step or the test runner
99+
# itself hangs. 90 min is well above any realistic run time.
100+
timeout-minutes: 90
96101
strategy:
97102
fail-fast: false
98103
matrix:

0 commit comments

Comments
 (0)