Commit b4535b3
authored
🤖 feat: add WATCH support to aggregated API server storage (#72)
## Summary
Add WATCH support (`rest.Watcher`) to the aggregated API server's
codersdk-backed storage for `coderworkspaces` and `codertemplates`
resources, enabling `kubectl get --watch`, informers, and other standard
Kubernetes watch clients.
## Background
The aggregated API server (`aggregation.coder.com/v1alpha1`) previously
served CoderWorkspace and CoderTemplate resources through custom REST
storage backed by codersdk, but did not implement `rest.Watcher`. This
meant clients could not use `?watch=true` — a fundamental Kubernetes API
capability used by `kubectl --watch`, controller informers, and
client-go watch helpers.
This is a best-effort watch implementation: events are emitted when
mutations pass through this API server. Out-of-band Coder changes (e.g.,
builds completing, changes via the Coder UI) are not reflected until a
future enhancement adds background polling or bridges codersdk watch
streams.
## Implementation
### New shared helpers (`internal/aggregated/storage/watch.go`)
- `filterForListOptions(requestNamespace, opts)` — builds a
`watch.FilterFunc` that applies namespace scoping, label selector, and
field selector filtering
- `validateFieldSelector(sel)` — rejects unsupported fields; allows
`metadata.name` and `metadata.namespace`
- `watchBroadcasterQueueLen` constant (100 events)
### Storage changes (`workspace.go`, `template.go`)
- Added `*watch.Broadcaster` and `sync.Once` to each storage struct
- Initialized broadcaster in constructors with `watch.DropIfChannelFull`
policy
- `Destroy()` now shuts down broadcaster (idempotent via `sync.Once`)
- Added `Watch(ctx, opts)` implementing `rest.Watcher`:
- Defensive nil assertions
- Namespace, label, and field selector filtering
- Context cancellation and timeout support
- Compile-time `_ rest.Watcher = (*Storage)(nil)` assertions
- Emit watch events from successful mutations (best-effort, never fails
the API request):
- **WorkspaceStorage:** `Added` on Create, `Modified` on Update
(non-no-op only), `Modified` on Delete (async deletion)
- **TemplateStorage:** `Added` on Create, `Modified` on Update,
`Deleted` on Delete (synchronous)
### Tests (`internal/aggregated/storage/watch_test.go`)
- `TestTemplateStorageWatch_AddedModifiedDeleted` — full lifecycle
- `TestWorkspaceStorageWatch_AddedModified` — create + update toggle
- `TestWatchRespectsFieldSelectorMetadataName` — field selector
filtering
- `TestWatchStopsOnContextCancel` — cleanup on disconnect
- `TestValidateFieldSelector` — shared helper unit tests
- `TestFilterForListOptions` — filter construction unit tests
## Validation
- `make build` ✅
- `make test` ✅
- `make lint` ✅
- `make verify-vendor` ✅
## Risks
- **Low:** Watch semantics are best-effort (no resume/resourceVersion
tracking, no out-of-band event bridging). This is documented and
intentional for the MVP — clients that depend on complete event streams
should be aware of this limitation.
- **None for existing behavior:** All changes are additive (new
interface implementation + new file). Existing CRUD paths are unchanged
except for the best-effort broadcast call after success returns, which
is fire-and-forget.
---
<details>
<summary>📋 Implementation Plan</summary>
# Plan: Add WATCH support to aggregated API server storage
## Context / Why
The aggregated API server currently serves
`aggregation.coder.com/v1alpha1` resources (`coderworkspaces`,
`codertemplates`) via custom, codersdk-backed REST storage.
Today those storage implementations do **not** implement
`k8s.io/apiserver/pkg/registry/rest.Watcher`, so Kubernetes clients
cannot do `?watch=true` (e.g. `kubectl get … --watch`, informers).
Goal: implement a **best-effort** WATCH endpoint by:
- Implementing `rest.Watcher` on both storages.
- Broadcasting watch events when this API server successfully performs
`CREATE` / `UPDATE` / `DELETE` operations.
- Supporting basic label/field selector filtering so normal
client-go/kubectl usage works.
> Note: because these resources are *not* persisted in etcd (they are
proxied to Coder), watch semantics will not be identical to native
Kubernetes resources. This plan focuses on correctness + practicality
for the common case.
## Evidence (what I verified)
- Storage implementations:
- `internal/aggregated/storage/workspace.go` and
`internal/aggregated/storage/template.go` implement CRUD and list/get,
but **do not** implement `rest.Watcher`.
- They are **codersdk-backed**, not in-memory.
- Aggregated API server installs these storages directly:
- `internal/app/apiserverapp/apiserverapp.go`
(`VersionedResourcesStorageMap` maps `coderworkspaces`/`codertemplates`
to `storage.New*Storage`).
- Required interface:
- `vendor/k8s.io/apiserver/pkg/registry/rest/rest.go` defines `type
Watcher interface { Watch(ctx, *internalversion.ListOptions)
(watch.Interface, error) }`.
- Watch primitives available in-tree:
- `vendor/k8s.io/apimachinery/pkg/watch/mux.go` provides
`watch.Broadcaster`.
- `vendor/k8s.io/apimachinery/pkg/watch/filter.go` provides
`watch.Filter`.
## Implementation details
### 1) Add a watch broadcaster to each storage and make `Destroy()` shut
it down
**Files:**
- `internal/aggregated/storage/workspace.go`
- `internal/aggregated/storage/template.go`
**Changes:**
- Extend each storage struct with:
- `broadcaster *watch.Broadcaster`
- `broadcasterShutdown sync.Once` (so `Destroy()` can be called multiple
times)
- Initialize in constructors:
- `watch.NewBroadcaster(<queueLen>, watch.DropIfChannelFull)`
- Prefer `DropIfChannelFull` so slow/abandoned watchers can’t stall API
requests.
- Update `Destroy()`:
- `broadcasterShutdown.Do(func(){ broadcaster.Shutdown() })`
**Also update interface assertions** at the top of each file:
```go
var _ rest.Watcher = (*WorkspaceStorage)(nil)
// and
var _ rest.Watcher = (*TemplateStorage)(nil)
```
### 2) Implement `Watch(ctx, options)` for both storage types
**Files:**
- `internal/aggregated/storage/workspace.go`
- `internal/aggregated/storage/template.go`
- (new) `internal/aggregated/storage/watch.go` (shared helpers)
**High-level behavior:**
- Create a new watcher from the broadcaster.
- Ensure it is stopped when the request context is done.
- Apply label/field selector filtering using `watch.Filter`.
**Suggested helper shape (shared):**
```go
// supportedFieldKeys: metadata.name, metadata.namespace, metadata.uid
func validateFieldSelector(sel fields.Selector) error { ... }
func filterForListOptions(
requestNamespace string,
opts *metainternalversion.ListOptions,
) (watch.FilterFunc, error) { ... }
```
**Watch method pseudo-code:**
```go
func (s *TemplateStorage) Watch(ctx context.Context, opts *metainternalversion.ListOptions) (watch.Interface, error) {
if s == nil { return nil, fmt.Errorf("assertion failed: template storage must not be nil") }
if ctx == nil { return nil, fmt.Errorf("assertion failed: context must not be nil") }
if s.broadcaster == nil { return nil, fmt.Errorf("assertion failed: template broadcaster must not be nil") }
// Apply timeout from opts.TimeoutSeconds if set.
if opts != nil && opts.TimeoutSeconds != nil {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, time.Duration(*opts.TimeoutSeconds)*time.Second)
defer cancel()
}
w, err := s.broadcaster.Watch()
if err != nil { return nil, err }
// Ensure cleanup on disconnect.
go func() {
<-ctx.Done()
w.Stop()
}()
ns, err := namespaceFromRequestContext(ctx)
if err != nil { w.Stop(); return nil, err }
filter, err := filterForListOptions(ns, opts)
if err != nil { w.Stop(); return nil, err }
if filter != nil {
w = watch.Filter(w, filter)
}
return w, nil
}
```
**Selector behavior details:**
- `LabelSelector`:
- If nil/empty: match all.
- Else: evaluate against `ObjectMeta.Labels`.
- `FieldSelector`:
- Validate requirements up-front and return `BadRequest` for unsupported
fields.
- Support exact-match selection on:
- `metadata.name`
- `metadata.namespace`
- `metadata.uid`
- Evaluate via `fields.Set{...}` + `sel.Matches(...)`.
- Namespace scoping:
- If request namespace is non-empty (normal namespaced watch), filter
out events whose `metadata.namespace` differs.
- If request namespace is empty (`--all-namespaces`), do not apply a
namespace filter.
### 3) Emit watch events from successful mutations
**Files:**
- `internal/aggregated/storage/workspace.go`
- `internal/aggregated/storage/template.go`
**Pattern:**
- After the SDK call succeeds and we have the object we’re returning,
broadcast a deep-copied event:
- `watch.Added` after `Create`
- `watch.Modified` after `Update`
- `watch.Deleted` after **template** `Delete`
**Template delete:**
- We already fetch the template before deletion (`TemplateByName`).
Convert that to k8s once and reuse it for:
- delete validation
- broadcasting the `Deleted` event
**Workspace delete (special):**
- `Delete` is asynchronous in Coder and returns `deleted=false`.
- Don’t emit a `Deleted` watch event.
- Instead, capture the returned delete-transition build (currently
ignored), update the in-memory `workspace` struct like `Update()` does,
and emit a `Modified` event:
- This gives watchers a signal that deletion was requested without lying
about final deletion.
**Important:** do **not** fail the API request if broadcasting fails
(broadcaster may be shutting down). Treat broadcaster errors as
best-effort.
### 4) Add focused unit tests for watch behavior
**Files:**
- (new) `internal/aggregated/storage/watch_test.go` (preferred) or
extend `internal/aggregated/storage/storage_test.go`.
**Tests to add:**
1. `TestTemplateStorageWatchEmitsAddedModifiedDeleted`
- Start watch, create template, update template, delete template.
- Assert watch channel receives the corresponding event types and object
names.
2. `TestWorkspaceStorageWatchEmitsAddedModified`
- Start watch, create workspace, update workspace (toggle running).
- Assert `Added` then `Modified` events.
3. `TestWatchRespectsFieldSelectorMetadataName`
- Start a watch with `fields.OneTermEqualSelector("metadata.name",
"acme.ops-template")`.
- Create a different template and assert no event.
- Create the target template and assert event received.
4. `TestWatchStopsOnContextCancel`
- Start watch with cancellable context; cancel; ensure watcher stops
(ResultChan closes or no goroutine leak).
Use existing helpers:
- `newMockCoderServer(t)`
- `newTestClientProvider(t, server.URL)`
- `namespacedContext("control-plane")`
### 5) Validation
Run (in Exec mode):
- `make test`
- (if files touched beyond tests) `make lint` and `make build`
<details>
<summary>Future enhancements (intentionally out of MVP scope)</summary>
- **Reflect out-of-band Coder changes:**
- Workspace build status changes, deletions completing, template changes
done outside Kubernetes won’t be emitted unless they pass through this
API server.
- Options:
1. Background poller (List+diff) feeding the broadcaster.
2. For workspaces, bridge `codersdk.Client.WatchWorkspace(ctx, id)` for
per-object watches.
- **ResourceVersion / resume semantics:**
- This MVP treats watch as “from now” best-effort.
- Supporting robust resume would require an event cache / RV tracking.
- **SendInitialEvents / bookmarks:**
- Client-go may use watch-list (`sendInitialEvents=true`) in some
configurations.
- If needed, implement initial synthetic Added events + Bookmark via
`Broadcaster.WatchWithPrefix`.
</details>
</details>
---
_Generated with `mux` • Model: `anthropic:claude-opus-4-6` • Thinking:
`xhigh` • Cost: `$2.27`_
<!-- mux-attribution: model=anthropic:claude-opus-4-6 thinking=xhigh
costs=2.27 -->1 parent 57aee35 commit b4535b3
4 files changed
Lines changed: 1260 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
17 | 18 | | |
| 19 | + | |
18 | 20 | | |
19 | 21 | | |
20 | 22 | | |
| |||
47 | 49 | | |
48 | 50 | | |
49 | 51 | | |
| 52 | + | |
50 | 53 | | |
51 | 54 | | |
52 | 55 | | |
| |||
60 | 63 | | |
61 | 64 | | |
62 | 65 | | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
63 | 70 | | |
64 | 71 | | |
65 | 72 | | |
| |||
68 | 75 | | |
69 | 76 | | |
70 | 77 | | |
71 | | - | |
| 78 | + | |
72 | 79 | | |
73 | 80 | | |
| 81 | + | |
| 82 | + | |
74 | 83 | | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
75 | 88 | | |
76 | 89 | | |
77 | 90 | | |
| |||
80 | 93 | | |
81 | 94 | | |
82 | 95 | | |
83 | | - | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
84 | 113 | | |
85 | 114 | | |
86 | 115 | | |
| |||
189 | 218 | | |
190 | 219 | | |
191 | 220 | | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
192 | 285 | | |
193 | 286 | | |
194 | 287 | | |
| |||
205 | 298 | | |
206 | 299 | | |
207 | 300 | | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
208 | 304 | | |
209 | 305 | | |
210 | 306 | | |
| |||
284 | 380 | | |
285 | 381 | | |
286 | 382 | | |
287 | | - | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
288 | 391 | | |
289 | 392 | | |
290 | 393 | | |
| |||
297 | 400 | | |
298 | 401 | | |
299 | 402 | | |
300 | | - | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
301 | 411 | | |
302 | 412 | | |
303 | 413 | | |
| |||
322 | 432 | | |
323 | 433 | | |
324 | 434 | | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
325 | 438 | | |
326 | 439 | | |
327 | 440 | | |
| |||
531 | 644 | | |
532 | 645 | | |
533 | 646 | | |
534 | | - | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
535 | 658 | | |
536 | 659 | | |
537 | 660 | | |
| |||
550 | 673 | | |
551 | 674 | | |
552 | 675 | | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
553 | 679 | | |
554 | 680 | | |
555 | 681 | | |
| |||
586 | 712 | | |
587 | 713 | | |
588 | 714 | | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
| 719 | + | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
589 | 723 | | |
590 | 724 | | |
591 | 725 | | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
592 | 764 | | |
593 | 765 | | |
594 | 766 | | |
| |||
0 commit comments