Skip to content

fix: Refuse to overwrite foreign objects from TrustStore reconciler - #707

Merged
dervoeti merged 2 commits into
mainfrom
fix/truststore-foreign-object-hijack
Jun 24, 2026
Merged

fix: Refuse to overwrite foreign objects from TrustStore reconciler#707
dervoeti merged 2 commits into
mainfrom
fix/truststore-foreign-object-hijack

Conversation

@dervoeti

@dervoeti dervoeti commented May 18, 2026

Copy link
Copy Markdown
Member

Description

TLDR: Write access on TrustStore allows write access for ConfigMap or Secret in the same namespace

The TrustStore reconciler writes its output ConfigMap or Secret using the TrustStore CR's own name and applies it via SSA with force=true. Combined with the operator's cluster-wide write permissions on ConfigMaps and Secrets, this lets any principal who can create a TrustStore in a namespace cause the operator to overwrite the contents of any pre-existing same-named ConfigMap or Secret in that namespace, stealing ownership and attaching a controller OwnerReference that links the hijacked object to the TrustStore CR for cascade delete.

A good showcase is the auto-published kube-root-ca.crt ConfigMap:
A tenant with create on truststores.secrets.stackable.tech in their namespace can create a TrustStore named kube-root-ca.crt, overwrite the CA bundle every pod in the namespace uses to verify in-cluster TLS, and cascade-delete that ConfigMap by deleting the TrustStore. This allows a) MITM attacks (hard because you need to modify DNS resolution of the Pods) and b) potentially crashing existing Pods on restart because kube-root-ca.crt is corrupted.

The same applies to any ConfigMap or Secret in the same namespace whose name the attacker can guess.

Fix

Before applying, look up the existing target. If the object already exists and is not owned by the current TrustStore, refuse the reconciliation.

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

  • Changes are OpenShift compatible
  • CRD changes approved
  • CRD documentation for all fields, following the style guide.
  • Helm chart can be installed and deployed operator works
  • Integration tests passed (for non trivial changes)
  • Changes need to be "offline" compatible
  • Links to generated (nightly) docs added
  • Release note snippet added

Reviewer

  • Code contains useful comments
  • Code contains useful logging statements
  • (Integration-)Test cases added
  • Documentation added or updated. Follows the style guide.
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Feature Tracker has been updated
  • Proper release label has been added
  • Links to generated (nightly) docs added
  • Release note snippet added
  • Add type/deprecation label & add to the deprecation schedule
  • Add type/experimental label & add to the experimental features tracker

@dervoeti
dervoeti force-pushed the fix/truststore-foreign-object-hijack branch 2 times, most recently from b3fc25f to 5b555d9 Compare May 18, 2026 13:01
@dervoeti
dervoeti force-pushed the fix/truststore-foreign-object-hijack branch from 5b555d9 to 940dde4 Compare May 18, 2026 13:02
sweb
sweb previously approved these changes May 18, 2026

@sweb sweb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm - but I primarily did this to dive into the code base, so grains of salt - I also tried out the review skill on this.

Comment thread rust/operator-binary/src/truststore_controller.rs Outdated
Comment thread rust/operator-binary/src/truststore_controller.rs Outdated
Comment thread rust/operator-binary/src/truststore_controller.rs Outdated
Comment thread rust/operator-binary/src/truststore_controller.rs
@dervoeti dervoeti moved this to Development: Waiting for Review in Stackable Engineering May 18, 2026
@dervoeti dervoeti self-assigned this May 18, 2026

@NickLarsenNZ NickLarsenNZ left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks ready to merge.

@NickLarsenNZ
NickLarsenNZ added this pull request to the merge queue Jun 19, 2026
@NickLarsenNZ NickLarsenNZ moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Jun 19, 2026
@sbernauer
sbernauer removed this pull request from the merge queue due to a manual request Jun 19, 2026
@NickLarsenNZ

Copy link
Copy Markdown
Member

Merged cancelled due to a comment from @sbernauer

IIRC he wanted to do a little bit more digging around it to see what other places of the SDP are "affected". Or if this is all over the place and we are only fixing the tip of the iceberg. In any case we likely want some docs comparing it to cert-manager and trust-manger etc.

@dervoeti

Copy link
Copy Markdown
Member Author

https://github.com/cert-manager/cert-manager has a similar problem but less severe (keys in Secrets/ConfigMaps are fixed and can't be chosen by an attacker), but https://github.com/cert-manager/trust-manager has the same problem. That does not mean we can't do it better though.

We have discovery ConfigMaps in other operators, but that problem is limited to ConfigMaps and the keys are fixed there as well. This problem in secret-operator is more severe since it allows overwriting ConfigMaps and Secrets and the keys can be chosen freely (at least not the content, but still a more severe problem, see the kube-root-ca.crt example in the initial comment).

For Listeners, a user can override/delete any Service in the namespace that he can create Listeners in. That is similar to the problem this PR fixes and could be fixed as well.

I did not find any similar problems.

We have three options:
a) merge this and create a PR for listener-operator as well (two local fixes)
b) put the guard into operator-rs so it does ownership checks in general when patching resources (one global fix)
c) leave everything as is and accept the risk

The real security benefit is probably not too high (having permissions to create TrustStores / Listeners is a requirement to exploit this) but it also should not break anything or make it much worse. I'm leaning towards option a) since it fixes the two worst problems and the blast radius is limited.

@sbernauer what's your opinion on this?

@sbernauer sbernauer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I let you decide, ship it :)

@dervoeti
dervoeti added this pull request to the merge queue Jun 24, 2026
Merged via the queue into main with commit 0a103eb Jun 24, 2026
12 checks passed
@dervoeti
dervoeti deleted the fix/truststore-foreign-object-hijack branch June 24, 2026 11:41
@dervoeti dervoeti moved this from Development: In Review to Development: Done in Stackable Engineering Jun 25, 2026
@lfrancke

Copy link
Copy Markdown
Member

Should this have a release note?

@lfrancke lfrancke moved this from Development: Done to Acceptance: In Progress in Stackable Engineering Jun 29, 2026
@dervoeti

Copy link
Copy Markdown
Member Author

Should this have a release note?

I don't think so, this should not affect customers in any way, it just strengthens the basic security of SDP a bit.

@lfrancke lfrancke moved this from Acceptance: In Progress to Done in Stackable Engineering Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

5 participants