|
| 1 | +# Backup Auth Secret Must Not Be Owned by Any Workspace |
| 2 | + |
| 3 | +**Status**: Accepted |
| 4 | +**Date**: 2026-05-11 |
| 5 | +**Deciders**: DevWorkspace Operator maintainers |
| 6 | +**Related Issue**: [CRW-10760](https://redhat.atlassian.net/browse/CRW-10760) |
| 7 | + |
| 8 | +## Context |
| 9 | + |
| 10 | +The backup system copies the registry authentication secret (e.g., quay.io credentials) from the operator namespace into each workspace namespace as `devworkspace-backup-registry-auth`. The original implementation set a Kubernetes controller ownerReference from this secret to the DevWorkspace that triggered the copy: |
| 11 | + |
| 12 | +```go |
| 13 | +controllerutil.SetControllerReference(workspace, desiredSecret, scheme) |
| 14 | +``` |
| 15 | + |
| 16 | +This was likely intended to clean up the secret when no workspaces need it anymore — standard Kubernetes garbage collection pattern. |
| 17 | + |
| 18 | +However, two properties of this secret make ownerReference-based lifecycle management incorrect: |
| 19 | + |
| 20 | +1. **The secret is a namespace singleton**: All workspaces in a namespace share the same `devworkspace-backup-registry-auth` secret, but a Kubernetes object can have only one controller owner. Whichever workspace's backup job ran last "wins" ownership. |
| 21 | + |
| 22 | +2. **The secret is needed after all workspaces are deleted**: The primary restore use case is creating a new workspace from a backup of a deleted one. If the auth secret is garbage-collected when the last workspace is deleted, the user cannot authenticate to the private registry to pull the backup image. |
| 23 | + |
| 24 | +**Bug observed (CRW-10760)**: When using quay.io (private registry) for backups, deleting a workspace caused backup entries to disappear from the Dashboard backup list for ALL workspaces in the namespace. The auth secret was garbage-collected, and the Dashboard could no longer query the registry. |
| 25 | + |
| 26 | +**Validated on CRC cluster** (DWO 0.40.1, quay.io/okurinny): |
| 27 | +- The `devworkspace-backup-registry-auth` secret was confirmed to have ownerReference to a single workspace (`nodejs`) |
| 28 | +- Deleting that workspace triggered K8s GC, removing the secret |
| 29 | +- The secret was not re-created by subsequent backup cycles (remaining workspaces already had recent backups) |
| 30 | +- Backup listing in the Dashboard failed for all workspaces |
| 31 | + |
| 32 | +## Decision |
| 33 | + |
| 34 | +**Remove the ownerReference from the backup registry auth secret.** The secret becomes a namespace-scoped resource with no ownership tie to any workspace. |
| 35 | + |
| 36 | +### What Changes |
| 37 | + |
| 38 | +- `pkg/secrets/backup.go`: Remove the `controllerutil.SetControllerReference()` call in `CopySecret()` |
| 39 | +- The secret is still created via `c.Create()` with `AlreadyExists` handling, just without an ownerReference |
| 40 | +- The restore path now resolves the operator namespace via `infrastructure.GetNamespace()` to copy the secret on demand when it is missing |
| 41 | + |
| 42 | +### What Doesn't Change |
| 43 | + |
| 44 | +- Per-workspace resources (job runner ServiceAccount, image-builder RoleBinding) retain their ownerReferences — their GC on workspace deletion is correct and expected |
| 45 | +- The backup Job itself retains its ownerReference (short-lived, TTL-cleaned) |
| 46 | +- The `CopySecret` function signature stays the same |
| 47 | + |
| 48 | +## Considered Alternatives |
| 49 | + |
| 50 | +### Alternative 1: Multi-owner references (non-controller) |
| 51 | + |
| 52 | +Add each workspace as a non-controller owner of the secret. K8s GC would only delete the secret when ALL owning workspaces are gone. |
| 53 | + |
| 54 | +**Rejected because**: |
| 55 | +- The secret must survive deletion of ALL workspaces (for restore) |
| 56 | +- Adds complexity to track and merge owner lists |
| 57 | +- Non-controller ownerReferences have subtle GC semantics |
| 58 | + |
| 59 | +### Alternative 2: Finalizer-based cleanup |
| 60 | + |
| 61 | +Remove ownerReference but add a cleanup mechanism (e.g., a controller that deletes the secret when the namespace has zero DevWorkspaces). |
| 62 | + |
| 63 | +**Rejected because**: |
| 64 | +- Adds complexity for a marginal benefit (one small secret in an otherwise-empty namespace) |
| 65 | +- Could race with restore operations (user creates a workspace from backup right after the last one is deleted) |
| 66 | +- Namespace deletion already cleans up all resources |
| 67 | + |
| 68 | +### Alternative 3: Per-workspace auth secrets |
| 69 | + |
| 70 | +Create a unique auth secret per workspace (e.g., `devworkspace-backup-registry-auth-{workspace-id}`). |
| 71 | + |
| 72 | +**Rejected because**: |
| 73 | +- Multiplies secrets unnecessarily (all contain the same credentials) |
| 74 | +- The restore path expects the predefined name `devworkspace-backup-registry-auth` |
| 75 | +- Still wouldn't survive workspace deletion for restore use case |
| 76 | + |
| 77 | +## Consequences |
| 78 | + |
| 79 | +### Positive |
| 80 | + |
| 81 | +1. **Backups survive workspace deletion**: Users can delete all workspaces and still restore from backups |
| 82 | +2. **No cross-workspace interference**: Deleting one workspace no longer affects other workspaces' backup capabilities |
| 83 | +3. **Simpler lifecycle**: No ownership tracking needed for a namespace-scoped singleton |
| 84 | + |
| 85 | +### Negative |
| 86 | + |
| 87 | +1. **Secret persists in empty namespaces**: If all workspaces are deleted and the user never restores, the auth secret remains until the namespace is deleted. This is a minor leak — one small secret per namespace. |
| 88 | + |
| 89 | +### Neutral |
| 90 | + |
| 91 | +1. **Existing secrets on upgraded clusters**: Secrets created by older DWO versions will retain their stale ownerReference until the next `CopySecret` call overwrites them (via `SyncObjectWithCluster`). In the worst case, one more GC event occurs before the fix takes effect. |
| 92 | + |
| 93 | +## References |
| 94 | + |
| 95 | +- `pkg/secrets/backup.go` — `CopySecret()` function |
| 96 | +- `controllers/backupcronjob/rbac.go` — Per-workspace SA/RoleBinding (unchanged) |
| 97 | +- `pkg/constants/metadata.go:204` — `DevWorkspaceBackupAuthSecretName` |
0 commit comments