Merge branch 'main' into feat/sites-47306-api-contract

Author: iuliag

.gitignore

@@ -21,6 +21,7 @@ admin-idp-p*.json
 # V1 to V2 migration scripts and test data (local only)
 scripts/
 !scripts/backfill-rum-config.mjs
+!scripts/serenity-rightsizing-sweep.mjs
 docs/index.html
 mcp-server.log
 # Scout MCP local config (per-developer, never commit)

CHANGELOG.md

@@ -1,3 +1,17 @@
+## [1.661.1](https://github.com/adobe/spacecat-api-service/compare/v1.661.0...v1.661.1) (2026-07-17)
+
+
+### Bug Fixes
+
+* Prompt Research prompts count from real per-topic total ([#2843](https://github.com/adobe/spacecat-api-service/issues/2843)) ([5f7f4af](https://github.com/adobe/spacecat-api-service/commit/5f7f4af946e50a824ffc3a5785a494819e989e51))
+
+# [1.661.0](https://github.com/adobe/spacecat-api-service/compare/v1.660.0...v1.661.0) (2026-07-17)
+
+
+### Features
+
+* **serenity:** rollout hardening — rightsizing sweep, observability, cross-container lock ADR (LLMO-6191) ([#2811](https://github.com/adobe/spacecat-api-service/issues/2811)) ([1d54a1a](https://github.com/adobe/spacecat-api-service/commit/1d54a1a93347b07a90c179faa7b54b825b8a2fe5)), closes [#2764](https://github.com/adobe/spacecat-api-service/issues/2764) [#2764](https://github.com/adobe/spacecat-api-service/issues/2764)
+
 # [1.660.0](https://github.com/adobe/spacecat-api-service/compare/v1.659.1...v1.660.0) (2026-07-16)
 
 

docs/decisions/007-cross-container-resource-lock.md

@@ -0,0 +1,107 @@
+# ADR-007: Cross-container serialization for the dynamic-allocation absolute-set race
+
+## Context
+
+The dynamic (JIT) Semrush AI resource allocator (PR #2764, `SERENITY_DYNAMIC_ALLOCATION`, default
+OFF) does an **absolute set** of a sub-workspace's resource `total` — it reads the child's current
+totals, computes a new one, and transfers it (`ensureAiHeadroom` and `releaseAiSurplus`, both in
+`src/support/serenity/resource-manager.js`). Two operations that read the same child concurrently
+and then each write an absolute value can clobber one another: the later write wins with a value
+computed from a now-stale read.
+
+`src/support/serenity/resource-lock.js` (`withResourceLock`) serializes same-child mutations via an
+in-process promise chain with a `LOCK_TIMEOUT_MS` (10s) safety valve. This PR (LLMO-6191) also wraps
+`releaseAiSurplus`'s one production call site (`markets-subworkspace.js`, the model-update seam) in
+the same lock, so **both** `ensureAiHeadroom` and `releaseAiSurplus` now serialize against each other
+for the same child, within one warm Lambda container. That closes the same-container half of the
+race. Across separate warm containers — the normal case for a Lambda-per-request API service — there
+is **no serialization at all**, so the race is fully open cross-container. This ADR is about closing
+that remaining gap (or deciding, explicitly, not to yet).
+
+Constraints on the options below: no new cloud infrastructure may be provisioned unilaterally by this
+PR (no new DynamoDB table, no new Redis/ElastiCache instance — see the LLMO-6191 scope guidance).
+This repo has **zero existing DynamoDB usage** (no `@aws-sdk/client-dynamodb` dependency) and no
+distributed-lock/conditional-write primitive was found in `@adobe/spacecat-shared` either.
+
+### Corrected failure-mode analysis (staff-engineer review)
+
+The absolute-set mechanic that CAUSES the race is also what BOUNDS it — this matters for how risky
+"do nothing yet" actually is:
+
+- **ensure-vs-ensure** (two concurrent top-ups on the same child): each call computes its own
+  `target = roundUpToBlock(...)`, ceiling-clamped (`resource-manager.js`, `ensureAiHeadroom`). The
+  last writer wins with **its own legitimate, ceiling-bounded target** — the final total is never
+  higher than `max(target_a, target_b)`. No ceiling breach, no unbounded over-write, no double-charge
+  of the parent pool. Worst case: the loser's top-up is silently absorbed and the next metered op
+  that finds itself still short just re-triggers `ensureAiHeadroom` — self-healing.
+- **ensure-vs-release** (an `ensureAiHeadroom` topping up a child racing a `releaseAiSurplus` lowering
+  the same child — now closed IN-PROCESS by this PR's lock wrap, but still open cross-container): a
+  stale `release` writing last can set `total` below the now-real `used` (a transient
+  **oversubscribed child**, rejecting the in-flight metered write until the next `ensure` heals it —
+  not silent data loss, but not "just re-tops-up" either). A stale `ensure` writing last after a
+  `release` already lowered the child can draw more from the master than its own advisory pool check
+  computed (bounded only by the master's terminal `422`, not by the advisory read).
+
+Neither interleaving causes data loss or a permanent ceiling breach. The corrected framing: **worst
+case is a transient oversubscription/over-draw window, self-healing on the next `ensureAiHeadroom`,
+never corruption or a stuck permanent state.**
+
+## Decision
+
+**Adopt Option C (accept the current risk band, observe, revisit on signal) for the initial ON
+rollout — conditional on the LLMO-6191 item-2 observability landing first**, not as an independent
+decision made in isolation. `TopUpLatencyMs`, `PoolFreeRatio`, and the `AllocationRejection`/
+`ReleaseOutcome` metrics (`src/support/serenity/allocation-metrics.js`) are the concrete signal this
+ADR's "revisit if it manifests" promise depends on — this status is NOT "Accepted, ship and forget."
+
+Options considered:
+
+**A — DynamoDB conditional write (new infra).** A lock-holder row per child workspace id, written
+with a `ConditionExpression` (only succeeds if unheld or the holder's TTL expired), TTL as the
+auto-expiry safety valve mirroring `LOCK_TIMEOUT_MS`. Correct, well-understood pattern. Cost: a new
+table, IAM policy, and an operational surface (TTL tuning, hot-partition risk on a busy child) this
+repo does not have today. Requires provisioning review — out of scope for this PR per the ticket's
+scope guidance.
+
+**B — Push the fix upstream (Semrush optimistic concurrency).** If the Semrush transfer API accepted
+an `expected_total` (compare-and-swap) parameter, a stale writer would get a rejection instead of
+silently clobbering, and no distributed lock would be needed at all — the upstream call becomes
+self-serializing. **Unverified**: whether the live gateway supports this is an open question, not
+confirmed against the tenant. If it does, this is the strongest option (no new infra, and it fixes
+the root cause rather than working around it). Action item: check with the Semrush/Project-Engine
+API owners before ruling this out permanently.
+
+**C — Accept + observe (recommended for now).** Ship with the current risk band: same-container race
+closed (this PR), cross-container race open but bounded to a transient, self-healing oversubscription
+window (never corruption, never a permanent ceiling breach — see the analysis above). Watch the
+item-2 telemetry (`TopUpLatencyMs`, `PoolFreeRatio`, `AllocationRejection`, `ReleaseOutcome`) for
+repeated-oversubscription symptoms (e.g. a `workspaceBusy` spike shortly after a top-up, or a
+`ReleaseOutcome` pattern suggesting a release clobbered a subsequent ensure) as the live signal of
+whether this actually manifests at current traffic. Revisit with Option A, B, or D if it does.
+
+**D — SQS FIFO with `MessageGroupId = childWorkspaceId` (SRE review addition).** This repo already
+speaks SQS (`src/support/sqs.js`); a FIFO queue's per-group ordering IS a cross-container
+serialization primitive without provisioning new infrastructure CLASS (an existing AWS primitive,
+not a new one). It would reframe the hot path from synchronous-locked to enqueue-and-settle, which
+also retires the fail-fast/504 tension `ensureAiHeadroom`'s hot path currently has to work around.
+Real costs: async top-up latency (the request would have to wait for a queue round-trip, or the
+metered write would have to happen optimistically before the top-up is confirmed — a different
+correctness contract than today's), a new consumer worker, and reordering the synchronous gate
+contract every metered handler currently relies on (`createHeadroomGuard.ensure` is awaited
+in-request today). Worth a real look if Option C's telemetry shows the race manifesting, but a bigger
+shape change than A or B.
+
+**E — Single-flight coalescing (SRE review addition, NOT cross-container).** Not a serialization
+primitive on its own — de-duplicating concurrent identical top-up requests within a container reduces
+contention but does not touch the cross-container case. Cheap, but only a partial mitigation; already
+effectively achieved in-container by `withResourceLock`.
+
+## Status
+
+Accepted, contingent on the item-2 observability metrics being live and watched (see LLMO-6191 PR).
+In-PR mitigation shipped now: `releaseAiSurplus`'s one production call site is wrapped in
+`withResourceLock` alongside `ensureAiHeadroom`, closing the same-container ensure/release race —
+see `src/support/serenity/handlers/markets-subworkspace.js`. The cross-container gap remains open,
+by design, per Option C above. Revisit this ADR (promote to Option A, B, or D) if the item-2 signals
+show the race manifesting at real traffic, or if the Semrush API owners confirm Option B is
+available — check that before committing engineering time to Option A's new infrastructure.

docs/runbooks/serenity-zombie-workspace-recovery.md

@@ -0,0 +1,149 @@
+# Runbook: Serenity zombie sub-workspace recovery (dynamic AI allocation)
+
+**Last validated:** 2026-07-14 (LLMO-6191 rollout-hardening).
+
+How to recognize and recover a Semrush sub-workspace that is stuck "not ready" after a
+partially-applied dynamic-allocation transfer, and how to read the paging signals for the JIT
+top-up allocator (`SERENITY_DYNAMIC_ALLOCATION`).
+
+---
+
+## When to use this
+
+- A brand's metered writes (create/publish prompts, add markets, model updates) are returning
+  `503 Sub-workspace is provisioning, retry` (`workspaceBusy`) repeatedly for the **same brand**,
+  well past a normal settle window.
+- `AllocationRejection{Reason=workspaceBusy}` or a `NotReadyRetry` streak (see
+  [Observability](#observability--paging-signals) below) is elevated for one workspace and not
+  clearing on its own.
+- A support ticket reports one brand permanently unable to add prompts/markets while other brands
+  are unaffected (scoped to one workspace — not a broad outage; see the release-outage runbook for
+  that case instead).
+
+## Prerequisites
+
+- **Access:** an IMS user token with standing on the affected org's Semrush workspace (same
+  requirement as any `/serenity/*` call — see `docs/serenity.md`; there is **no service-account
+  path** to Semrush in this repo). `mysticat auth token --ims`.
+- **The brand's ids:** the SpaceCat brand id and its `semrushSubWorkspaceId` (read via
+  `GET /v2/orgs/:org/brands/:brandId`, or directly from the `brands` table).
+- Familiarity with `src/support/serenity/resource-manager.js` (`ensureAiHeadroom`,
+  `releaseAiSurplus`) and `workspace-lifecycle.js` (`pollUntilCreated`) helps but is not required to
+  follow the steps below.
+
+## TL;DR fast path
+
+```bash
+export API_BASE=https://spacecat.experiencecloud.live/api/ci
+export IMS=$(mysticat auth token --ims)
+
+# 1. Confirm the workspace's live status
+curl -s -H "Authorization: Bearer ${IMS}" \
+  "${API_BASE}/v2/orgs/${ORG_ID}/brands/${BRAND_ID}/serenity/markets" | jq .
+
+# 2. Retry the metered op that's failing (the transfer is idempotent — a retry after settling
+#    finds it already applied and proceeds normally). Most "zombie" cases self-heal on the very
+#    next customer-facing retry once the async workspace lock actually clears.
+
+# 3. If it does NOT clear after a few minutes, escalate — this repo has no admin endpoint to force
+#    a workspace out of "not ready" (see Step 3 below).
+```
+
+---
+
+## Step 1 — Confirm the symptom, not just the alarm
+
+`workspaceBusy` (503) is the allocator's **expected, retryable** signal when a transfer hits the
+Semrush async "workspace not ready" lock (`isWorkspaceNotReady`, `src/support/serenity/errors.js`).
+A single 503 is normal — the transfer is absolute + idempotent, so the client/UI is expected to
+retry. This runbook is for when the SAME workspace keeps returning it well past a normal settle
+window (minutes, not seconds), i.e. the async lock itself appears stuck.
+
+Grep Splunk for the specific workspace (see `docs/serenity.md`'s Observability section for the
+index/sourcetype):
+
+```spl
+index=dx_aem_engineering sourcetype=dx_aem_sites_spacecat_backend_<env> service=api-service
+  "SERENITY_ALLOC" "<subWorkspaceId>"
+```
+
+Look for a repeated `SERENITY_ALLOC transfer never cleared workspace-not-ready` line
+(`resource-manager.js`, `transferAndSettle`) or repeated `SERENITY_ALLOC workspace not ready on
+transfer — returning 503` (`transferOnce`, the fail-fast hot path) for the SAME workspace id across
+multiple, separated requests.
+
+## Step 2 — Rule out an in-flight partial transfer (the actual "zombie" cause)
+
+A sub-workspace can be left "not ready" when a transfer was applied upstream but the
+Adobe/Semrush-side async settle never completed — the write went out, but the workspace's public
+status keeps reporting a transitional state. Confirm live status:
+
+```bash
+curl -s -H "Authorization: Bearer ${IMS}" \
+  "${API_BASE}/v2/orgs/${ORG_ID}/brands/${BRAND_ID}/serenity/markets" | jq '.[].status'
+```
+
+If markets report `status` other than the expected live value, or the brand's activation state
+(`GET /v2/orgs/:org/brands/:brandId`) shows `pending` when you expect `active`, the workspace is
+likely still mid-settle upstream, not a bug in this codebase.
+
+## Step 3 — Attempt recovery
+
+There is **no admin/automation endpoint in this repo** to force a stuck workspace's state — every
+`/serenity/*` call is a normal customer-facing operation gated on a live IMS user token (see
+`docs/serenity.md`'s "IMS-user only" caveat). Recovery options, in order:
+
+1. **Wait and retry the original failing operation.** The transfer is idempotent
+   (`transferOnce`/`transferAndSettle`, `resource-manager.js`) — a retry once the async lock clears
+   upstream simply finds the workspace already at the intended total and proceeds. Most cases
+   resolve within the `LOCK_TIMEOUT_MS`/poll-attempt windows already built into the allocator
+   (`DEFAULT_POLL`, 12 attempts × 1s, plus `NOT_READY_RETRIES` backoff — see `resource-manager.js`).
+2. **Re-run `ensureAiHeadroom` for the workspace manually** (e.g. via a one-off invocation in a
+   Node REPL against the target env, using `createSerenityTransport` the same way
+   `scripts/serenity-rightsizing-sweep.mjs` does) if you need to confirm the transfer settles
+   outside of a live customer request. This performs no destructive action — it only re-reads and,
+   if still short, re-tops-up.
+3. **If the workspace remains stuck for longer than ~30 minutes**, this is very likely an
+   upstream (Semrush) issue, not something this codebase can resolve — escalate to the Semrush/
+   Project-Engine API owners with the workspace id and the Splunk trace from Step 1. There is no
+   decommission-and-recreate path for a sub-workspace short of the destructive
+   `POST /serenity/deactivate` (which deletes every project in it — see `docs/serenity.md`'s
+   Activate/deactivate caveats) followed by a fresh `activate`; treat that as a last resort, and
+   confirm with the brand owner first since it is data-destructive.
+
+## Observability — paging signals
+
+See `src/support/serenity/allocation-metrics.js` for the full metric catalog
+(`Mysticat/SerenityAllocation` CloudWatch namespace). The two classes that matter for THIS runbook:
+
+| Metric | Dashboard-only or pager-worthy | Why |
+|---|---|---|
+| `AllocationRejection{Reason=orgPoolExhausted\|brandAiLimit}` | Dashboard-only | Expected under normal load on a small pool — not a bug. |
+| `AllocationRejection{Reason=workspaceBusy}` | **Pager-worthy** | The transfer never cleared the async lock — JIT top-up itself is degraded, not just short on quota. Sustained/repeated for one workspace is this runbook's trigger. |
+| `NotReadyRetry` | **Pager-worthy** if a run exhausts all retries (correlate with the `workspaceBusy` counter, which fires exactly when a retry run exhausts) | A rising rate without exhaustion is just gateway latency; exhaustion is the actionable signal. |
+| `ReleaseOutcome{Reason=requires-decommission}` | Dashboard-only, but track the trend | A standing pool-leak signal — surplus that can never be reclaimed short of workspace delete. Not urgent per-occurrence, but a rising trend means the rightsizing sweep (`scripts/serenity-rightsizing-sweep.mjs`) or a decommission pass is due. |
+| `ReleaseOutcome{Reason=error}` / `AllocationRejection` generally | Dashboard-only | Expected fleet noise on a live allocator. |
+
+**Alerting caveat:** this repo has no alerting-as-code file to declare a CloudWatch/Coralogix alarm
+in — the metrics above are emitted (EMF → CloudWatch), but wiring an actual alarm/pager rule for
+`AllocationRejection{Reason=workspaceBusy}` and retry-exhaustion is a **manual step** an operator
+must still take in the CloudWatch/Coralogix console (or via `mcp__coralogix__manage_alerts` if
+you're doing it from an agent session) — this PR does not invent a new alerting pipeline to do it
+automatically, per LLMO-6191's scope guidance. Do this before turning `SERENITY_DYNAMIC_ALLOCATION`
+on for real traffic.
+
+## Command quick-reference
+
+```bash
+# check a brand's live market/status
+curl -s -H "Authorization: Bearer ${IMS}" \
+  "${API_BASE}/v2/orgs/${ORG_ID}/brands/${BRAND_ID}/serenity/markets" | jq .
+
+# Splunk trace for one workspace
+index=dx_aem_engineering sourcetype=dx_aem_sites_spacecat_backend_<env> service=api-service
+  "SERENITY_ALLOC" "<subWorkspaceId>"
+
+# dry-run the rightsizing sweep against one brand (also a safe way to probe a workspace's state)
+POSTGREST_URL=... SEMRUSH_IMS_TOKEN=... node scripts/serenity-rightsizing-sweep.mjs \
+  --dry-run --org-ids ${ORG_ID} --brand-ids ${BRAND_ID}
+```

docs/serenity.md

@@ -299,6 +299,30 @@ Expected fields in the structured log payload:
 - outbound auth: `Authorization: Bearer ...` (the token itself is redacted by the platform)
 - the IMS sub of the caller in the `actor` field
 
+## Dynamic AI resource allocation — operations (LLMO-6191)
+
+The JIT top-up allocator (`SERENITY_DYNAMIC_ALLOCATION`, default OFF — see
+`src/support/serenity/dynamic-allocation-active.js`) has its own operational surface, separate from
+the request-path proxy documented above:
+
+- **Metrics/SLIs:** `src/support/serenity/allocation-metrics.js` emits CloudWatch EMF metrics
+  (namespace `Mysticat/SerenityAllocation`) — pool-free ratio, top-up latency, rejection/retry/
+  release-outcome counters, and the hot-path (topped-up vs not) ratio. See that file's module doc
+  for the full catalog and the pager-worthy/dashboard-only split.
+- **Zombie-workspace recovery:** see
+  [`docs/runbooks/serenity-zombie-workspace-recovery.md`](./runbooks/serenity-zombie-workspace-recovery.md)
+  for diagnosing and recovering a sub-workspace stuck `workspaceBusy` after a partially-applied
+  transfer, and for the alerting/paging guidance.
+- **Rightsizing sweep:** `scripts/serenity-rightsizing-sweep.mjs` is a one-time backfill that
+  lowers already-carved sub-workspaces (brands onboarded before the JIT allocator shipped) down to
+  their actual usage, using `releaseAiSurplus` as the reclaim primitive. Run `--dry-run` first —
+  see the script's own header comment for full usage and the auth caveat (requires an operator
+  IMS token; there is no service-account path to Semrush in this repo).
+- **Cross-container serialization:** `src/support/serenity/resource-lock.js` only serializes
+  same-container contention. The cross-container gap and the options considered for closing it are
+  recorded in
+  [`docs/decisions/007-cross-container-resource-lock.md`](./decisions/007-cross-container-resource-lock.md).
+
 ## Dev environment smoke tests
 
 After the api-service feature branch deploys to dev, exercise the surface against `https://spacecat.experiencecloud.live/api/ci`.