✨ Concurrent E2E Test Execution#2675
Conversation
Takes advantage of changes made to isolate test runs to execute as many tests in parallel as possible. For tests that must be run serially, the @serial tag has been added to the beginning of relevant feature file(s). Signed-off-by: Daniel Franz <dfranz@redhat.com> Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
✅ Deploy Preview for olmv1 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Pull request overview
This PR updates the E2E Godog test runner to take advantage of scenario isolation (from #2651) by running most scenarios in parallel while reserving tagged scenarios for a serial pass, and then printing a combined failure summary.
Changes:
- Split E2E execution into two Godog runs: parallel (excluding
@Serial) followed by serial (@Serialonly), with aggregated failure output. - Tag TLS E2E feature(s) with
@Serialso they run only in the serial phase. - Reduce
test-experimental-e2etimeout from 20m to 10m.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| test/e2e/features_test.go | Runs E2E suite in parallel + serial phases and prints an aggregated failure summary. |
| test/e2e/features/tls.feature | Marks TLS feature as @Serial to exclude it from the parallel phase. |
| Makefile | Adjusts experimental E2E timeout setting. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| parallelOpts := defaultOpts | ||
| parallelOpts.Concurrency = 100 | ||
| parallelOpts.Tags = "~@Serial" | ||
| parallelOpts.Randomize = -1 | ||
| // Write to both stdout (live) and buffer (for summary) | ||
| parallelOpts.Output = io.MultiWriter(os.Stdout, ¶llelBuf) | ||
| // run tests concurrently | ||
| scParallel := godog.TestSuite{ | ||
| TestSuiteInitializer: InitializeSuite, | ||
| ScenarioInitializer: InitializeScenario, | ||
| Options: ¶llelOpts, | ||
| }.Run() | ||
|
|
||
| fmt.Println("End of parallel run - beginning serial tests") | ||
|
|
||
| serialOpts := defaultOpts | ||
| serialOpts.Concurrency = 1 | ||
| serialOpts.Tags = "@Serial" | ||
| // Write to both stdout (live) and buffer (for summary) |
There was a problem hiding this comment.
parallelOpts.Tags and serialOpts.Tags overwrite any --godog.tags provided on the command line, which breaks the documented workflow for running a subset of scenarios by tag (see test/e2e/README.md). Consider combining the user-supplied tag expression (if any) with the Serial filter (e.g., "() && ~@serial" and "() && @serial"), rather than replacing it entirely.
There was a problem hiding this comment.
Actually, I'll re-work this to instead explicitly use the tags provided by CLI when they are there and reduce the concurrency value to 1, unless also provided.
| // Write to both stdout (live) and buffer (for summary) | ||
| parallelOpts.Output = io.MultiWriter(os.Stdout, ¶llelBuf) | ||
| // run tests concurrently |
There was a problem hiding this comment.
The MultiWriter for live+buffer output writes to os.Stdout directly, which bypasses defaultOpts.Output (currently colors.Colored(os.Stdout)), so enabling colors via flags would no longer have any effect. Use defaultOpts.Output (or parallelOpts.Output before overriding) as the stdout writer to preserve configured formatting.
| test-experimental-e2e: export INSTALL_DEFAULT_CATALOGS := false | ||
| test-experimental-e2e: PROMETHEUS_VALUES := helm/prom_experimental.yaml | ||
| test-experimental-e2e: E2E_TIMEOUT := 20m | ||
| test-experimental-e2e: E2E_TIMEOUT := 10m |
There was a problem hiding this comment.
This PR reduces test-experimental-e2e timeout from 20m to 10m, but the PR description focuses on parallelizing execution rather than changing timeout expectations. Given earlier work increased this timeout to accommodate dynamic per-scenario registry/catalog setup, please confirm with CI timings (or keep 20m) to avoid reintroducing flaky timeouts.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2675 +/- ##
==========================================
+ Coverage 67.99% 68.11% +0.12%
==========================================
Files 144 144
Lines 10573 10573
==========================================
+ Hits 7189 7202 +13
+ Misses 2865 2854 -11
+ Partials 519 517 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
Description
Takes advantage of changes made to isolate test runs (#2651) to execute as many tests in parallel as possible. For tests that must be run serially, the
@Serialtag has been added to the beginning of relevant feature file(s).Tests are now executed in two steps: parallel followed by serial. Since this creates two distinct test outputs, the final output has been modified to combine any step failures into one smaller output at the end, while still maintaining live output to stdout. Ideally this could be formatted into a nicer looking markdown output in the test summary that we already output, but I'm saving that for a future PR.
Reviewer Checklist