Make acceptance CI test matrix parallel by kswap · Pull Request #5295 · hashicorp/consul-k8s

kswap · 2026-05-07T09:14:08Z

Changes proposed in this PR

Core feature

Added --crd-install-flag support to the acceptance test framework and TerminatingGateway types.

CI: Test matrix parallelization

Rewrote matrix generator (Python → Go) with run-filter support, enabling individual Test* functions to run on independent parallel runners.
Split 9 monolithic table-driven tests into top-level functions across partitions, connect-lifecycle, config-entries, sameness, consul-dns, connect, peering, terminating-gateway, and wan-federation.
Added num-clusters declarations for partitions, peering, wan-federation, and api-gateway to skip unnecessary cluster creation on CNI runners.
Moved API gateway external server tests to a new dedicated api-gateway-external package.

CI: Reliability fixes

Port-forward reconnect: TCP readiness probe prevents ECONNREFUSED races after tunnel restart.
Vault auth: retry loop waiting for K8s 1.24+ async ServiceAccount token population, fixing TestVault_Partitions flake.
Config-entries: bumped controller reconciliation retry from 5s to 60s, fixing tproxy flake.

How I've tested this PR

How I expect reviewers to test this PR

Checklist

Tests added
CHANGELOG entry added

PCI review checklist

I have documented a clear reason for, and description of, the change I am making.
If applicable, I've documented a plan to revert these changes if they require more than reverting the pull request.
If applicable, I've documented the impact of any changes to security controls.

Examples of changes to security controls include using new access control methods, adding or removing logging pipelines, etc.

santoshpulluri · 2026-05-12T18:05:21Z

        workflow: test.yml
        repo: hashicorp/consul-k8s-workflows
-        ref: main
+        ref: acceptance-test-crd-parallel


Why is this changed from main branch to acceptance-test-crd-install-flag branch ?

It is done to test with the consul-k8s-workflows branch temporarily which hass dependent changes

i hope this will be changed to main after workflows are merged to main in k8s-workflows ?

kswap · 2026-05-17T13:13:57Z

/rerun

…cated runners

…nal, update helm_cluster_test

… CLI tests

Splits 7 monolithic table-driven acceptance tests (38 cases total) into individual top-level Test* functions so each gets its own dedicated CI runner via the existing bin-packing sharding system. Drops the test ceiling from 55 min (TestPartitions_Connect, 6 cases) to ~11 min (TestWANFederationFailover per case). Updates test-timings.json with per-case estimates so the matrix generator can bin-pack correctly from the first run.

…ailure

…patibility

…lity

…API CRDs

…ce race

…pod readiness check

…for wan-federation

…ortServices (4 cases); add num-clusters:2 for partitions

…d files

…m retry

…-clusters for api-gateway and peering

…rning from kind-cni-calico

…s 2+ contexts)

anandmukul93 · 2026-05-22T07:53:34Z

why was there a need to modify these files so much ? what benefit have we gotten with exposing test type to each test case now ? can we retry at case level or still package level only ? if not so why to make so many changes to the file.

anandmukul93 · 2026-05-22T08:20:15Z

i believe this is part of ci workflows . then it could be in .github/scripts folder rather than build-support in control-plane.

anandmukul93 · 2026-05-22T08:32:38Z

+			}
+		}
+
+		discovered := discoverTests(repoRoot, pkg)


rename variables sensibly . suggested discoveredTestsInPackage.

kswap requested review from a team as code owners May 7, 2026 09:14

kswap added pr/no-changelog PR does not need a corresponding .changelog entry pr/no-backport signals that a PR will not contain a backport label labels May 7, 2026

santoshpulluri reviewed May 12, 2026

View reviewed changes

kswap changed the title ~~Acceptance test crd install flag~~ Make acceptance CI test matrix parallel May 18, 2026

kswap added 23 commits May 20, 2026 11:28

acceptance: parallelize tests and split connect/api-gateway into dedi…

911c2f1

…cated runners

feat: add global.installCRDs flag and pre-apply CRDs in test suite

e1e78f8

test: add t.Parallel() to acceptance tests and installCRDs bats coverage

8d6003e

ci: point workflow dispatch to acceptance-test-crd-parallel

2ff547d

fix: add missing StaticClientName and retryCheck to api-gateway-exter…

21dd550

…nal, update helm_cluster_test

ci: retrigger after artifact fix

a87a8ce

fix: add tls+meshGateway flags and filter CRD-only docs in installCRDs

efee6c2

fix: remove t.Parallel() from top-level acceptance tests

e3bf351

feat: time-weighted test sharding via bin-packing

6d60230

fix: use consul/consul release+namespace for CRD pre-install to match…

e31edfd

… CLI tests

reduce helm install retry count from 30 to 3 to prevent 2h hangs on f…

06f664f

…ailure

fix: install CRDs on all cluster contexts not just primary

8192684

fix: add helm ownership labels to pre-installed CRDs for CLI test com…

13c4813

…patibility

fix: resolve all golangci-lint issues for Go 1.26 toolchain compatibi…

3a8fdb5

…lity

fix: label all pre-installed CRDs by name including external gateway …

1867321

…API CRDs

fix: resolve Vault tunnel reuse and API Gateway GatewayClass acceptan…

7e00e23

…ce race

fix: revert vault tunnel fix, increase gateway scaling timeouts, fix …

cc5fb81

…pod readiness check

fix vault portforward reuse bug in initAndUnseal retry loop

745db33

fix: handle pod-gone race in ProxyLifecycleShutdown grace period loop

7826bc3

fix: remove invalid govet 'inline' analyzer from golangci config

4ce181f

fix: check kubectl output for NotFound in grace period pod-gone race

88412ed

ci: retrigger after ignore_failed_clean fix

e446228

kswap added 21 commits May 20, 2026 11:30

Fix flaky CI: portforward readiness probe, num-clusters matrix field …

bbf0b3a

…for wan-federation

docs: add CI reliability and performance improvements design doc

aa9a410

Split TestPartitions_Sync (6 cases) and TestPartitions_Connect_Multip…

43d8ebe

…ortServices (4 cases); add num-clusters:2 for partitions

pr cleanup: remove doc, revert godot-only comment changes in unrelate…

fbd58b2

…d files

rewrite generate_test_matrix.py in Go; drop Python dependency

291f7a1

revert global.installCRDs; keep only helm install retry fix (30->3)

e791a81

ci: retrigger after setup-go fix in workflows

5d4d6e9

ci: retrigger after vault flake

343fc64

ci: retrigger after infra flakes (port-forward, 404, pod eviction)

be102f8

ci: retrigger after v1.30.0 runner flakes

3560d52

ci: split config-entries/connect-lifecycle/sameness tests and fix hel…

6c83ab7

…m retry

fix: remove duplicate var err declaration in runProxyLifecycleShutdown

0ec32c5

ci: replace sleep 30/20 in kind-cni-calico with kubectl wait; set num…

59716cf

…-clusters for api-gateway and peering

fix: wait for calico-node DaemonSet to exist and roll out before retu…

a0c3850

…rning from kind-cni-calico

ci: retrigger after vault 403 and HPA race flakes

cdbf17a

fix: retry waiting for SA token population in ConfigureAuthMethod

2cb1ec2

fix: increase retry timeout for config entry propagation checks

8835485

docs: update CI_IMPROVEMENTS with Phase 3 changes and Phase 4 targets

3336341

docs: rewrite CI_IMPROVEMENTS as single consolidated improvement doc

33507b4

revert: remove CI_IMPROVEMENTS.md from PR

8f44ebd

fix: change api-gateway num-clusters to 2 (multi-cluster flag require…

ade0809

…s 2+ contexts)

kswap force-pushed the acceptance-test-crd-install-flag branch from 7b3ed94 to ade0809 Compare May 20, 2026 06:01

kswap and others added 4 commits May 20, 2026 12:10

charts: add comment documenting installK8sNetworkingCRDs behavior

3d17643

ci: update test-timings.json from run 26146024961

c3bfec0

ci: update test-timings.json from run 26148744372

72cf2c8

ci: update test-timings.json from run 26151707436

b9bd2ea

anandmukul93 reviewed May 22, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make acceptance CI test matrix parallel#5295

Make acceptance CI test matrix parallel#5295
kswap wants to merge 48 commits into
mainfrom
acceptance-test-crd-install-flag

kswap commented May 7, 2026 •

edited

Loading

Uh oh!

santoshpulluri May 12, 2026

Uh oh!

kswap May 18, 2026 •

edited

Loading

Uh oh!

anandmukul93 May 22, 2026

Uh oh!

kswap commented May 17, 2026

Uh oh!

anandmukul93 May 22, 2026 •

edited

Loading

Uh oh!

anandmukul93 May 22, 2026

Uh oh!

anandmukul93 May 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

kswap commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes proposed in this PR

How I've tested this PR

How I expect reviewers to test this PR

Checklist

PCI review checklist

Uh oh!

santoshpulluri May 12, 2026

Choose a reason for hiding this comment

Uh oh!

kswap May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anandmukul93 May 22, 2026

Choose a reason for hiding this comment

Uh oh!

kswap commented May 17, 2026

Uh oh!

anandmukul93 May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anandmukul93 May 22, 2026

Choose a reason for hiding this comment

Uh oh!

anandmukul93 May 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kswap commented May 7, 2026 •

edited

Loading

kswap May 18, 2026 •

edited

Loading

anandmukul93 May 22, 2026 •

edited

Loading