feat(observability): admin-gated per-request Server-Timing via X-OS-Debug-Timing (#2408) (#3163)

* feat(observability): admin-gated per-request Server-Timing via X-OS-Debug-Timing (#2408)

Adds the per-request gating from the perf-tuning design so an operator can pull
a single request's `Server-Timing` breakdown on a live environment without
turning the header on for every user.

Until now perf-tuning was global-only (`serverTiming` option / `OS_SERVER_TIMING`),
which discloses internal phase durations — a mild backend-fingerprinting surface
— to every caller. This wires up the second gating path the issue describes:

- **Disclosure gate** (`@objectstack/observability`): a request-scoped
  `AsyncLocalStorage` gate, separate from the pure `PerfTiming` collector so the
  collector keeps its "only measures, never decides to emit" invariant. Pinned
  to a `Symbol.for` registry key like the collector store so the middleware
  (which seeds it) and the dispatcher (which opens it) share one store across
  ESM/CJS module copies.
- **Hono middleware**: always registered unless `serverTiming: false`. Runs the
  collector when timing is global OR the request carries `X-OS-Debug-Timing: 1`;
  emits the header only when the gate is open. Zero collector overhead (one
  header read) when neither applies.
- **Dispatcher**: after resolving the execution context, opens the gate for
  admin/service/system principals (`isSystem`, `principalKind` service/system,
  posture PLATFORM_ADMIN/TENANT_ADMIN). Ordinary callers get no header even if
  they send the debug header.
- **Env alias**: `OS_PERF_TIMING=1` now also enables global mode, matching the
  issue's naming; `OS_SERVER_TIMING=true` still works.

Tests cover the gate primitive, the middleware's per-request/global/hard-off
paths, and the admin predicate. Existing global-mode behavior is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011upHdyr5AnNc6dWAu63qxD

* docs(observability): document the admin-gated per-request Server-Timing path (#2408)

Updates the Server-Timing section for the new gating: the global mode
(`serverTiming` / `OS_SERVER_TIMING` / `OS_PERF_TIMING`) plus the per-request
`X-OS-Debug-Timing: 1` path that returns timing only to an admin/service
identity, and the `serverTiming: false` hard-off. Keeps the go-live checklist
in sync.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011upHdyr5AnNc6dWAu63qxD

* chore(changeset): perf-timing per-request gating (#2408)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011upHdyr5AnNc6dWAu63qxD

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: os-zhuang

.changeset/perf-timing-per-request-gating.md

@@ -0,0 +1,29 @@
+---
+"@objectstack/observability": minor
+"@objectstack/plugin-hono-server": minor
+"@objectstack/runtime": minor
+---
+
+feat(observability): admin-gated per-request `Server-Timing` via `X-OS-Debug-Timing` (#2408)
+
+Perf-tuning mode was previously global-only (`serverTiming` option /
+`OS_SERVER_TIMING`), which discloses internal phase durations — a mild
+backend-fingerprinting surface — to every caller. This adds the per-request
+gating path from the design so an operator can pull a single request's
+`Server-Timing` breakdown on a live environment without turning the header on
+for everyone.
+
+- **observability**: a request-scoped disclosure gate (`runWithPerfDisclosure`,
+  `allowPerfDisclosure`, `isPerfDisclosureAllowed`, `PerfDisclosureGate`) kept
+  separate from the pure `PerfTiming` collector and pinned to its own
+  `Symbol.for` store so the middleware and dispatcher share it across module
+  copies.
+- **plugin-hono-server**: the Server-Timing middleware is registered by default
+  (unless `serverTiming: false`). It runs the collector when timing is global
+  **or** the request sends `X-OS-Debug-Timing: 1`, and emits the header only
+  when the gate is open. `OS_PERF_TIMING=1` now also enables global mode.
+- **runtime**: after resolving the execution context, the dispatcher opens the
+  gate for admin/service/system principals, so ordinary callers never receive
+  the header even if they send the debug header.
+
+Existing global-mode behavior is unchanged.

docs/OBSERVABILITY.md

@@ -238,27 +238,45 @@ usual culprit behind an inexplicably slow list), 18ms in **3** business hooks,
 and 7ms serializing the response. `db` and `hooks` are aggregates — one member
 carrying the summed duration and the event count, not one member per query.
 
-This is **off by default**: the header discloses internal phase durations,
-which is helpful for profiling but also lets a caller fingerprint the backend.
-Treat it as a perf-tuning toggle you flip in staging (or briefly in production
-behind an allowlist), not a default-on header.
+The header discloses internal phase durations, which is helpful for profiling
+but also lets a caller fingerprint the backend, so disclosure is **gated**.
+There are two ways to turn it on:
 
-Enable it on the Hono server plugin:
+**Global** — every response carries the header. Flip this in staging, or
+briefly on a production environment under active investigation:
 
 ```ts
 new HonoServerPlugin({ serverTiming: true });
 ```
 
 …or, for the default `os serve` server (which constructs the plugin for you),
-via the environment:
+via the environment (`OS_PERF_TIMING=1` and the older `OS_SERVER_TIMING=true`
+are equivalent):
 
 ```bash
-OS_SERVER_TIMING=true os serve
+OS_PERF_TIMING=1 os serve
 ```
 
-When enabled, every response carries `total` (the whole request, measured by
-an outer middleware) plus the sub-phases the request path records out of the
-box:
+**Per-request** — available by default (unless hard-disabled below), with **no
+redeploy**: the caller sends the request header
+
+```
+X-OS-Debug-Timing: 1
+```
+
+and the `Server-Timing` header comes back **only** when the request resolves an
+**admin/service identity** (a platform/tenant admin, a service token, or an
+internal system call). An ordinary user who sends the header gets nothing back —
+they can never pull timings, so this is safe to leave available on a live
+environment. This is the path to reach for when diagnosing "why is *this* request
+slow?" against a running environment.
+
+To hard-disable both paths (no middleware registered at all), set
+`serverTiming: false` explicitly.
+
+When timing is emitted, every response carries `total` (the whole request,
+measured by an outer middleware) plus the sub-phases the request path records
+out of the box:
 
 | Member | Recorded by | Meaning |
 |:---|:---|:---|
@@ -316,5 +334,7 @@ countServerTiming('db', queryMs, 'queries'); // → db;dur=<sum>;desc="<n> queri
 - [ ] Log records include `requestId` field; cross-checked one against the
       response `X-Request-Id` header.
 - [ ] Alerts wired: error rate, p95 latency per route.
-- [ ] (Optional) `Server-Timing` verified in DevTools when `serverTiming` /
-      `OS_SERVER_TIMING=true` is enabled, and confirmed **absent** by default.
+- [ ] (Optional) `Server-Timing` verified in DevTools with global mode
+      (`serverTiming: true` / `OS_PERF_TIMING=1`) on, confirmed **absent** for a
+      normal request, and confirmed the per-request `X-OS-Debug-Timing: 1` header
+      returns timing **only** to an admin/service caller.

packages/observability/src/index.ts

@@ -52,5 +52,9 @@ export {
     startServerTiming,
     measureServerTiming,
     countServerTiming,
+    runWithPerfDisclosure,
+    allowPerfDisclosure,
+    isPerfDisclosureAllowed,
     type ServerTimingMark,
+    type PerfDisclosureGate,
 } from './perf-timing.js';

packages/observability/src/perf-timing.test.ts

@@ -10,6 +10,10 @@ import {
     startServerTiming,
     measureServerTiming,
     countServerTiming,
+    runWithPerfDisclosure,
+    allowPerfDisclosure,
+    isPerfDisclosureAllowed,
+    type PerfDisclosureGate,
 } from './perf-timing.js';
 
 describe('formatServerTiming', () => {
@@ -208,3 +212,61 @@ describe('ambient collector', () => {
         expect(t.marks().map((m) => m.name)).toContain('after-await');
     });
 });
+
+describe('disclosure gate', () => {
+    it('isPerfDisclosureAllowed() is false with no active gate', () => {
+        expect(isPerfDisclosureAllowed()).toBe(false);
+    });
+
+    it('allowPerfDisclosure() is a no-op with no active gate (does not throw)', () => {
+        allowPerfDisclosure();
+        expect(isPerfDisclosureAllowed()).toBe(false);
+    });
+
+    it('reflects the seeded state inside runWithPerfDisclosure', () => {
+        const closed: PerfDisclosureGate = { allowed: false };
+        runWithPerfDisclosure(closed, () => {
+            expect(isPerfDisclosureAllowed()).toBe(false);
+        });
+        const open: PerfDisclosureGate = { allowed: true };
+        runWithPerfDisclosure(open, () => {
+            expect(isPerfDisclosureAllowed()).toBe(true);
+        });
+    });
+
+    it('allowPerfDisclosure() opens a closed gate the caller still holds', async () => {
+        const gate: PerfDisclosureGate = { allowed: false };
+        await runWithPerfDisclosure(gate, async () => {
+            await new Promise((r) => setTimeout(r, 1));
+            allowPerfDisclosure(); // after an await — same ALS scope
+        });
+        // The caller reads its own reference once the scope settles.
+        expect(gate.allowed).toBe(true);
+    });
+
+    it('leaves the gate closed when disclosure is never granted', async () => {
+        const gate: PerfDisclosureGate = { allowed: false };
+        await runWithPerfDisclosure(gate, async () => {
+            await new Promise((r) => setTimeout(r, 1));
+        });
+        expect(gate.allowed).toBe(false);
+    });
+
+    it('pins the gate store to a global-registry symbol (shared across module copies)', () => {
+        const key = Symbol.for('@objectstack/observability:perf-disclosure-gate');
+        expect((globalThis as Record<symbol, unknown>)[key]).toBeDefined();
+    });
+
+    it('is independent of the timing collector scope', () => {
+        // A collector can be active without a gate, and vice versa.
+        const t = new PerfTiming();
+        runWithPerfTiming(t, () => {
+            expect(isPerfDisclosureAllowed()).toBe(false);
+        });
+        const gate: PerfDisclosureGate = { allowed: true };
+        runWithPerfDisclosure(gate, () => {
+            expect(currentPerfTiming()).toBeUndefined();
+            expect(isPerfDisclosureAllowed()).toBe(true);
+        });
+    });
+});

packages/observability/src/perf-timing.ts

@@ -283,3 +283,64 @@ export async function measureServerTiming<T>(
 export function countServerTiming(name: string, dur: number, unit?: string): void {
     store.getStore()?.count(name, dur, unit);
 }
+
+// --- Disclosure gate (WHO may see the timing) -------------------------
+
+/**
+ * Per-request disclosure gate — the policy counterpart to the collector.
+ *
+ * The {@link PerfTiming} collector only MEASURES; whether the measured
+ * `Server-Timing` header is returned to the client is a separate decision. When
+ * perf-tuning is turned on GLOBALLY (env flag / plugin option) the operator has
+ * opted the whole environment in, so the gate opens up front and every response
+ * carries the header. When it is turned on PER-REQUEST — the caller sends an
+ * `X-OS-Debug-Timing` header — the header must stay withheld until the request
+ * proves an admin/service identity: phase durations are a mild
+ * backend-fingerprinting surface, so an ordinary user must never be able to pull
+ * them just by sending a header. The request path flips the gate open with
+ * {@link allowPerfDisclosure} once it has resolved a privileged principal.
+ *
+ * Keeping this out of {@link PerfTiming} preserves the collector's invariant
+ * ("it only measures, it never decides whether to emit").
+ */
+export interface PerfDisclosureGate {
+    /** Whether the collected timing may be disclosed to the client. */
+    allowed: boolean;
+}
+
+/**
+ * The disclosure gate lives in its OWN global-registry-pinned
+ * `AsyncLocalStorage`, for the same cross-module-copy reason as the collector
+ * store above: the middleware seeds the gate and the dispatcher (a different
+ * package, possibly a different module copy) flips it open — both must see the
+ * one store.
+ */
+const GATE_KEY = Symbol.for('@objectstack/observability:perf-disclosure-gate');
+const globalGate = globalThis as unknown as Record<symbol, AsyncLocalStorage<PerfDisclosureGate> | undefined>;
+const gateStore: AsyncLocalStorage<PerfDisclosureGate> =
+    globalGate[GATE_KEY] ?? (globalGate[GATE_KEY] = new AsyncLocalStorage<PerfDisclosureGate>());
+
+/**
+ * Run `fn` with `gate` as the ambient disclosure gate for the async call chain.
+ * The caller keeps its reference to `gate` and reads `gate.allowed` after `fn`
+ * settles to decide whether to emit the header.
+ */
+export function runWithPerfDisclosure<T>(gate: PerfDisclosureGate, fn: () => T): T {
+    return gateStore.run(gate, fn);
+}
+
+/**
+ * Open the ambient disclosure gate — the request has proven it may see its own
+ * `Server-Timing` header (admin/service identity). A no-op when no gate is
+ * active (perf-tuning off, or already-global mode with no gate to flip), so the
+ * call site stays branch-free.
+ */
+export function allowPerfDisclosure(): void {
+    const g = gateStore.getStore();
+    if (g) g.allowed = true;
+}
+
+/** Whether the ambient disclosure gate is open. `false` when none is active. */
+export function isPerfDisclosureAllowed(): boolean {
+    return gateStore.getStore()?.allowed ?? false;
+}

packages/plugins/plugin-hono-server/src/hono-plugin.test.ts

@@ -210,36 +210,40 @@ describe('HonoServerPlugin', () => {
         });
 
         it('should disable CORS when cors option is false', async () => {
+            corsConfigCapture.last = undefined;
+
             const plugin = new HonoServerPlugin({
                 cors: false
             });
 
             await plugin.init(context as PluginContext);
 
-            const serverInstance = (HonoHttpServer as any).mock.instances[0];
-            const rawApp = serverInstance.getRawApp();
-
-            // CORS middleware should NOT be registered
-            expect(rawApp.use).not.toHaveBeenCalled();
+            // CORS middleware must NOT be configured. (Assert on the CORS config,
+            // not the raw `use` count: the perf-timing middleware registers its
+            // own `use('*')` by default to catch the `X-OS-Debug-Timing` header.)
+            expect(corsConfigCapture.last).toBeUndefined();
         });
 
         it('should disable CORS when CORS_ENABLED env is false', async () => {
             const originalEnv = process.env.OS_CORS_ENABLED;
             process.env.OS_CORS_ENABLED = 'false';
+            corsConfigCapture.last = undefined;
 
-            const plugin = new HonoServerPlugin();
-            await plugin.init(context as PluginContext);
-
-            const serverInstance = (HonoHttpServer as any).mock.instances[0];
-            const rawApp = serverInstance.getRawApp();
-
-            expect(rawApp.use).not.toHaveBeenCalled();
-
-            // Restore environment
-            if (originalEnv !== undefined) {
-                process.env.OS_CORS_ENABLED = originalEnv;
-            } else {
-                delete process.env.OS_CORS_ENABLED;
+            try {
+                const plugin = new HonoServerPlugin();
+                await plugin.init(context as PluginContext);
+
+                // CORS not configured — see the note above re: the perf-timing
+                // middleware's own `use('*')`.
+                expect(corsConfigCapture.last).toBeUndefined();
+            } finally {
+                // Restore environment even if the assertion fails, so a leaked
+                // `OS_CORS_ENABLED=false` can't disable CORS in later tests.
+                if (originalEnv !== undefined) {
+                    process.env.OS_CORS_ENABLED = originalEnv;
+                } else {
+                    delete process.env.OS_CORS_ENABLED;
+                }
             }
         });