[CC-35895] e2e/operator: add encryption provider interface and GCP advanced test infrastructure by shreyaskm623 · Pull Request #627 · cockroachdb/helm-charts

shreyaskm623 · 2026-05-04T05:17:28Z

Introduces an encryption.Provider interface to abstract KMS and secret setup behind infra-level provider types, and adds GCP-specific test infrastructure support including version-gated PCR connection URI generation.

JIRA : https://cockroachlabs.atlassian.net/browse/CC-35895

Copilot

Pull request overview

This PR refactors the operator E2E test harness to abstract encryption-at-rest setup behind a provider interface (so tests don’t manually generate/store keys) and extends infrastructure support for GCP, including CRDB-version-gated PCR external connection URI generation.

Changes:

Centralize provider resolution (PROVIDER) and wire an encryption.Provider into the region/provider factory flow.
Move encryption key/secret setup into a dedicated tests/e2e/operator/encryption package and update advanced encryption tests to use provider-driven overrides/secret creation.
Add GCP KMS-backed encryption test infrastructure and gate PCR URI generation on the CRDB version (convert-url vs encode-uri).

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/e2e/operator/singleRegion/cockroachdb_single_region_e2e_test.go	Switch to centralized provider resolution; provider-scoped test runner cleanup.
tests/e2e/operator/singleRegion/cockroachdb_single_region_advanced_features_test.go	Update encryption-at-rest flows (enable/disable/rotate) to use provider-driven overrides and secret creation; adjust PCR validation inputs.
tests/e2e/operator/region.go	Add encryption provider plumbing and encryption secret setup during install; add PCR URI version gating and primary cluster-domain override.
tests/e2e/operator/multiRegion/cockroachdb_multi_region_e2e_test.go	Switch to centralized provider resolution.
tests/e2e/operator/multiRegion/cockroachdb_multi_region_advanced_features_test.go	Update multi-region encryption test to use provider-driven encryption region config.
tests/e2e/operator/infra/provider.go	Add `ResolveProvider` and provider→region encryption provider wiring in the factory.
tests/e2e/operator/infra/local.go	Implement encryption provider interface for local providers (file-based/UNKNOWN_KEY_TYPE).
tests/e2e/operator/infra/gcp.go	Implement GCP KMS-backed encryption provider (keyring/key creation, encrypt, secret creation).
tests/e2e/operator/encryption/types.go	Introduce provider interface + helpers for generating keys and creating secrets for file-based vs CMEK providers.
tests/e2e/migrate/helm_chart_to_cockroach_enterprise_operator_test.go	Use centralized provider resolution for operator install region selection.

Comments suppressed due to low confidence (1)

tests/e2e/operator/singleRegion/cockroachdb_single_region_e2e_test.go:63

The provider teardown only calls cloudProvider.TeardownInfra; it does not trigger any encryption-provider cleanup even though Region stores an encryptionCleanupFunc. To avoid leaking KMS/test resources, call providerRegion.CleanupEncryptionInfra() (or equivalent) as part of this cleanup block.

		// Use t.Cleanup for guaranteed cleanup even on test timeout/panic.
		t.Cleanup(func() {
			t.Logf("Starting infrastructure cleanup for provider: %s", provider)
			cloudProvider.TeardownInfra(t)
			t.Logf("Completed infrastructure cleanup for provider: %s", provider)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+
+	out, err := exec.Command("kubectl", cmdArgs...).CombinedOutput()
+	if err != nil {
+		return fmt.Errorf("kubectl %s: %w\nOutput: %s", strings.Join(args[:2], " "), err, string(out))


+	// oldCmekCredentialsSecretName is required for the operator to set OLD_CMEK_PLATFORM
+	// and OLD_GCP_SERVICE_ACCOUNT_KEY, which tells the init container to KMS-decrypt
+	// the old key instead of writing the raw ciphertext to old_store.key.


Copilot

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

tests/e2e/operator/singleRegion/cockroachdb_single_region_e2e_test.go:63

The provider-level cleanup only calls cloudProvider.TeardownInfra, but encryption infra cleanup closures captured via Region (CleanupEncryptionInfra) are never invoked. To avoid leaking KMS/IAM test resources (and to keep the cleanup API meaningful), call providerRegion.CleanupEncryptionInfra() in both the t.Cleanup block and the immediate failure cleanup path.

		// Use t.Cleanup for guaranteed cleanup even on test timeout/panic.
		t.Cleanup(func() {
			t.Logf("Starting infrastructure cleanup for provider: %s", provider)
			cloudProvider.TeardownInfra(t)
			t.Logf("Completed infrastructure cleanup for provider: %s", provider)

+
+func (r *Region) EncryptionOverridesFromProvider() map[string]interface{} {
+	if r.encryptionProvider == nil {
+		return nil



-	// Wait for crd to be installed.
+	// Wait for crdbclusters CRD to serve v1beta1 before proceeding.
 	_, _ = retry.DoWithRetryE(t, "wait-for-crd", 60, time.Second*5, func() (string, error) {


+func SetupEncryptionSecrets(t *testing.T, provider Provider, kubectlOptions *k8s.KubectlOptions, clusterRegion string) error {
+	return SetupEncryptionSecretsWithName(t, provider, kubectlOptions, clusterRegion, "cmek-key-secret")
+}


Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

-		require.NoError(t, err)
-		require.True(t, len(secretSize) > 0, "Secret StoreKeyData should be >0")
-		t.Logf("Created encryption secret %s with size: %d bytes", encryptionSecretName, len(secretSize))
+		err = encryption.SetupEncryptionSecretsWithName(t, r.encryptionProvider, kubectlOptions, cluster, DefaultEncryptionSecret)


shreyaskm623 self-assigned this May 4, 2026

shreyaskm623 force-pushed the nishanth/operator-advanced-tests branch 2 times, most recently from ef82b74 to 6df483d Compare May 8, 2026 09:02

shreyaskm623 force-pushed the shreyaskm/gcp-operator-advanced-test-infra branch from 0326060 to 89e9a9c Compare May 8, 2026 09:49

shreyaskm623 force-pushed the nishanth/operator-advanced-tests branch 4 times, most recently from e210753 to 6ab0067 Compare May 12, 2026 12:47

shreyaskm623 force-pushed the shreyaskm/gcp-operator-advanced-test-infra branch 6 times, most recently from b1ddc9b to 90f3b64 Compare May 13, 2026 15:06

Base automatically changed from nishanth/operator-advanced-tests to master May 14, 2026 09:49

shreyaskm623 force-pushed the shreyaskm/gcp-operator-advanced-test-infra branch 4 times, most recently from d8a51b1 to 1a6687c Compare May 18, 2026 09:16

shreyaskm623 changed the title ~~operator: Adds advanced tests support for the GCP environment~~ e2e/operator: add encryption provider interface and GCP advanced test infrastructure May 18, 2026

shreyaskm623 force-pushed the shreyaskm/gcp-operator-advanced-test-infra branch 4 times, most recently from 4d14a02 to 1a6687c Compare May 18, 2026 15:34

shreyaskm623 marked this pull request as ready for review May 18, 2026 16:59

shreyaskm623 requested review from NishanthNalluri, Copilot, nameisbhaskar and pritesh-lahoti May 19, 2026 04:57

Copilot started reviewing on behalf of shreyaskm623 May 19, 2026 04:58 View session

Copilot AI reviewed May 19, 2026

View reviewed changes

shreyaskm623 requested a review from Copilot May 19, 2026 05:04

Copilot started reviewing on behalf of shreyaskm623 May 19, 2026 05:05 View session

Copilot AI reviewed May 19, 2026

View reviewed changes

nameisbhaskar reviewed May 19, 2026

View reviewed changes

Comment thread tests/e2e/operator/encryption/types.go

shreyaskm623 requested a review from Copilot May 20, 2026 07:37

Copilot started reviewing on behalf of shreyaskm623 May 20, 2026 07:37 View session

Copilot AI reviewed May 20, 2026

View reviewed changes

shreyaskm623 force-pushed the shreyaskm/gcp-operator-advanced-test-infra branch from 6e2b463 to c019669 Compare May 20, 2026 12:36

shreyaskm623 changed the title ~~e2e/operator: add encryption provider interface and GCP advanced test infrastructure~~ [CC-35895] e2e/operator: add encryption provider interface and GCP advanced test infrastructure May 20, 2026

shreyaskm623 added 4 commits May 20, 2026 22:26

Infrastructure changes to support GCP advanced tests

0e37b99

fixes and refactoring changes

3b9417d

adds the comment changes

7c133b5

updated images for the testing purposes

7c1ef66

shreyaskm623 force-pushed the shreyaskm/gcp-operator-advanced-test-infra branch from c019669 to 7c1ef66 Compare May 20, 2026 16:56

shreyaskm623 added 2 commits May 20, 2026 23:01

minor fix

ae9e60c

e2e/operator: Fix GCP single-region and multi-region test infrastructure

e388d5a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CC-35895] e2e/operator: add encryption provider interface and GCP advanced test infrastructure#627

[CC-35895] e2e/operator: add encryption provider interface and GCP advanced test infrastructure#627
shreyaskm623 wants to merge 6 commits into
masterfrom
shreyaskm/gcp-operator-advanced-test-infra

shreyaskm623 commented May 4, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

shreyaskm623 commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shreyaskm623 commented May 4, 2026 •

edited

Loading