Skip to content

fix: prevent test DB leakage from TestShell_BeforeNode#21916

Closed
Fletch153 wants to merge 1 commit intodevelopfrom
fix/test-db-leakage-beforenode
Closed

fix: prevent test DB leakage from TestShell_BeforeNode#21916
Fletch153 wants to merge 1 commit intodevelopfrom
fix/test-db-leakage-beforenode

Conversation

@Fletch153
Copy link
Copy Markdown
Collaborator

Summary

  • Root cause: NewApp().Before() unconditionally overwrites shell.Config with a fresh config using the default pgx driver, even when tests pre-set a config with the txdb driver for DB isolation. This causes BeforeNode() to open real database connections, leaking keystore entries (encrypted_key_rings) to the shared test database.
  • Symptom: Test_CSAKeyStore_E2E fails intermittently because it finds unexpected CSA keys leaked by TestShell_BeforeNode — causing PRs to be ejected from the merge queue.
  • Fix: Guard config creation in app.Before() with a nil check so pre-set test configs are preserved. Migrate TestShell_BeforeNode and TestShell_RunNode_WithBeforeNode to heavyweight.FullTestDBV2 for proper DB isolation.

Root Cause Trace

TestShell_BeforeNode
  → configtest.NewGeneralConfig() → DriverTxWrappedPostgres ✓
  → cmd.NewApp(&shell) → app.Before(c)
    → var opts GeneralConfigOpts{}   // fresh, empty
    → opts.New()                     // defaults → DriverPostgres (pgx)
    → s.Config = cfg                 // OVERWRITES test config ✗
  → shell.BeforeNode(c)
    → pg.NewLockedDB(cfg.Database()) // uses pgx, not txdb
    → BuildBeholderAuth() → EnsureKey() → safeAddKey()
    → CSA key written to encrypted_key_rings with pgx (LEAKS)

Changes

File Change
core/cmd/app.go Guard opts.New() + s.Config = cfg with if s.Config == nil — preserves pre-set test configs
core/cmd/shell_local_test.go Migrate TestShell_BeforeNode and TestShell_RunNode_WithBeforeNode to heavyweight.FullTestDBV2 for dedicated test databases

Why heavyweight?

The subtests are ordered: "correct password" creates a key ring, "incorrect password" verifies decryption fails with the wrong password. With txdb, each subtest gets an isolated transaction and can't see the other's data. heavyweight.FullTestDBV2 gives a dedicated database with pgx where subtests naturally share state, while the dedicated DB is dropped after the test — no leakage.

Test plan

  • TestShell_BeforeNode — all 3 subtests pass
  • TestShell_RunNode_WithBeforeNode — all 2 subtests pass
  • Test_CSAKeyStore_E2E — "initializes with an empty state" passes (the flaking test)
  • encrypted_key_rings count = 0 after test run (no leakage)
  • go vet ./core/cmd/... — no issues
  • CI go_core_tests passes

Related: #21504 (Erik's WIP fix for the same issue)

@Fletch153 Fletch153 requested review from a team as code owners April 8, 2026 15:31
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 8, 2026

👋 Fletch153, thanks for creating this pull request!

To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team.

Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks!

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 8, 2026

I see you updated files related to core. Please run make gocs in the root directory to add a changeset as well as in the text include at least one of the following tags:

  • #added For any new functionality added.
  • #breaking_change For any functionality that requires manual action for the node to boot.
  • #bugfix For bug fixes.
  • #changed For any change to the existing functionality.
  • #db_update For any feature that introduces updates to database schema.
  • #deprecation_notice For any upcoming deprecation functionality.
  • #internal For changesets that need to be excluded from the final changelog.
  • #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
  • #removed For any functionality/config that is removed.
  • #updated For any functionality that is updated.
  • #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 8, 2026

✅ No conflicts with other open PRs targeting develop

@trunk-io
Copy link
Copy Markdown

trunk-io Bot commented Apr 8, 2026

Static BadgeStatic BadgeStatic BadgeStatic Badge

Failed Test Failure Summary Logs
TestORM Logs ↗︎

View Full Report ↗︎Docs

@Fletch153 Fletch153 marked this pull request as draft April 8, 2026 18:43
NewApp().Before() unconditionally overwrites shell.Config with a fresh
config using the default pgx driver, even when tests pre-set a config
with the txdb driver for isolation. This causes BeforeNode to open real
database connections, leaking keystore entries (encrypted_key_rings) to
the shared test database. Concurrent tests like Test_CSAKeyStore_E2E
then find unexpected keys and fail.

Guard config creation in app.Before() with a nil check so pre-set test
configs are preserved. Migrate TestShell_BeforeNode and
TestShell_RunNode_WithBeforeNode to heavyweight.FullTestDBV2 for proper
DB isolation between subtests while preventing leakage to other tests.
@Fletch153 Fletch153 force-pushed the fix/test-db-leakage-beforenode branch from da148b0 to 79eec0b Compare April 8, 2026 18:50
@Fletch153 Fletch153 closed this Apr 8, 2026
@Fletch153 Fletch153 deleted the fix/test-db-leakage-beforenode branch April 8, 2026 19:16
@cl-sonarqube-production
Copy link
Copy Markdown

Quality Gate failed Quality Gate failed

Failed conditions
B Maintainability Rating on New Code (required ≥ A)
5.4% Technical Debt Ratio on New Code (required ≤ 4%)

See analysis details on SonarQube

Catch issues before they fail your Quality Gate with our IDE extension SonarQube IDE SonarQube IDE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant