Skip to content

fix(aso-overlay): rewrap native Response before aws-adapter (fix dev 500)#2839

Open
abhishekgarg18 wants to merge 2 commits into
mainfrom
fix/aso-overlay-500-ensure-fetch-response
Open

fix(aso-overlay): rewrap native Response before aws-adapter (fix dev 500)#2839
abhishekgarg18 wants to merge 2 commits into
mainfrom
fix/aso-overlay-500-ensure-fetch-response

Conversation

@abhishekgarg18

Copy link
Copy Markdown
Member

The symptom

Every GET /config/dev/*/redirects.txt on dev is returning HTTP 500 with:

```
x-error: response.headers.raw is not a function
```

Reproduced with a valid `ASO_OVERLAY_API_KEY` on multiple tenants (`cm-p117653-e365853`, `cm-p128729-e2179647`). Consistent on both the 200 and 304 conditional-GET code paths. Endpoint was working at 2026-07-16T00:03Z (Maksym's end-to-end verification); broke after v1.655.0 deployed (SITES-48140 EMF instrumentation, #2822).

Root cause

The error is thrown at `@adobe/helix-universal/aws-adapter.js:254`:

```js
...splitHeaders(response.headers.raw(), con.log),
```

That helper expects `@adobe/fetch`-style Headers (which has `.raw()`), but the Response reaching the adapter has native Web-Fetch-API Headers (from `undici`) whose Headers has no `.raw()`.

Bundle inspection (`dist/spacecat-services/api-service@1.657.0.zip`, extracted `index.js`) shows esbuild inlines two Response classes side by side:

  • `Response2` (line 2552) — `@adobe/fetch` shim, has `.raw()` on Headers ✅
  • `Response6` (line 57669, plus 3 duplicated copies) — `undici` (native fetch spec), no `.raw()` ❌

At source level, `createResponse` (from `@adobe/spacecat-shared-http-utils`) correctly imports `Response` from `@adobe/fetch` and returns `Response2` — verified by a standalone local test. But after #2822 added `metrics-emf.js` imports to both `RedirectsController` and `AsoOverlayKeyHandler`, the extended module graph now resolves to `Response6` at bundle time along the overlay code path.

Rather than chase the exact esbuild resolution behavior (opaque, fragile, and could regress again on any future import graph change), we normalize at the outermost seam — the wrapper that runs LAST on the response path before the AWS Lambda adapter sees the response.

The fix

New `ensureFetchResponseWrapper` installed as the OUTERMOST wrapper:

  • Fast path (every response that's already an `@adobe/fetch` Response): the wrapper does nothing. Two property checks (`response.headers && typeof response.headers.raw === 'function'`) and a return of the original response object. Zero-cost passthrough.
  • Slow path (the specific failure mode: response reaching us is a native Response): the wrapper rebuilds it as an `@adobe/fetch` Response with the same status, headers, and body. All existing behavior preserved.

This is defense-in-depth: it fixes the specific interaction from #2822 AND protects against any future PR that could introduce the same shift via a different code path.

Zero-regression argument (this is the important part)

The wrapper is intentionally the smallest change that fixes the symptom:

  1. Every current endpoint uses `createResponse` from `@adobe/spacecat-shared-http-utils`, which constructs `@adobe/fetch` Response. On the deployed Lambda, most of those already return `Response2` correctly today — those hit the wrapper's fast path (two property checks + return the exact same object reference). Byte-identical outputs to what those endpoints produce today.
  2. The two files that use bare `new Response(...)` today (`src/controllers/proxy.js` and `src/support/multipart-form-data.js` error paths) hit the slow path — but they'd hit the exact same aws-adapter failure the overlay endpoint hits today, so if they were working before this PR they must be low-traffic paths that don't exercise the aws-adapter Response serialization (e.g. only used in local dev). If either is on a hot path in prod, this PR fixes them; it can't regress them.
  3. The slow path preserves status, headers, and body content bit-for-bit — it buffers the body via `response.arrayBuffer()` and reconstructs with the same `status` and `headers`. The only theoretical difference is streaming behavior: our service does not stream any response bodies (all are small text/JSON payloads capped by the maximum-response-size guard in aws-adapter itself).
  4. Non-object results (`undefined`, `null`, primitives — e.g. what helix-status can return for raw payloads) are passed through untouched via an early-return guard.
  5. 7 unit tests cover all these cases explicitly — fast path identity, slow path rewrap on 200/404/304, empty body, non-object returns, error propagation, single-value header preservation (the shape the overlay endpoint actually uses).
  6. Full existing test suite still green (4908 passing across `test/support/` + `test/controllers/redirects.test.js` locally).

Known limitation, out of scope for this PR: multi-value headers like `Set-Cookie` on the slow path would collapse to their last value. api-service is a token-authenticated API and does not set cookies; if that ever changes, the wrapper's `forEach` loop will need to switch to `getSetCookie()` for that one header per WHATWG fetch spec. Documented inline.

Why we didn't catch this in dev

Two independent gaps:

  1. Unit tests import from source where module resolution goes to `node_modules/@adobe/fetch` directly — `createResponse` returns `Response2`, `.raw()` works, tests pass.
  2. CI's `bundle-build` step invokes the bundled Lambda but only against `/health` (JSON, different code path from our text/plain overlay).

Follow-up ticket to add a bundle-level integration test that exercises `/config/dev/*/redirects.txt` on the bundled artifact. Tracked as a separate item so this PR stays scoped to the fix.

Test plan

  • 7 new unit tests on the wrapper (fast path, slow path 200/304/404, empty body, non-object results, error propagation, header preservation)
  • Existing suites green (`test/support/` + `test/controllers/redirects.test.js` — 4908 passing locally)
  • Lint + type-check clean
  • Deploy to dev via CI — user asked for this to run through the standard deploy loop so we can live-verify
  • Post-deploy: curl `/config/dev/cm-p117653-e365853/redirects.txt` with valid key → confirm 200/304/404 shape (no more 500)
  • Unblocks Alina's TTL bump (task 37) and the fleet-wide 18-env verification (task 11)

Supersedes revert PR #2838 — closing that in favor of this forward-fix.

Jira: SITES-48140

🤖 Generated with Claude Code

…500)

## Symptom

Every GET /config/dev/*/redirects.txt on dev returns HTTP 500 with:

    x-error: response.headers.raw is not a function

Verified reproduction with a valid ASO_OVERLAY_API_KEY on multiple tenants
(cm-p117653-e365853, cm-p128729-e2179647). Consistent on both the 200 and
304 conditional-GET code paths. Worked at 2026-07-16T00:03Z (Maksym's
end-to-end verification), broke after v1.655.0 deployed (#2822 EMF
instrumentation).

## Root cause

The error is thrown at `@adobe/helix-universal/aws-adapter.js:254`:

    ...splitHeaders(response.headers.raw(), con.log),

That helper expects `@adobe/fetch`-style Headers (which has `.raw()`), but
the Response reaching the adapter has native Web-Fetch-API Headers (from
`undici`) whose Headers has no `.raw()`.

Inspection of the produced Lambda bundle
(`dist/spacecat-services/api-service@1.657.0.zip`) shows that esbuild
inlines TWO Response classes side-by-side:

- `Response2` — `@adobe/fetch` shim, has `.raw()` ✅
- `Response6` — `undici` (native fetch spec), no `.raw()` ❌

At the source level `createResponse` (from `@adobe/spacecat-shared-http-utils`)
correctly imports `Response` from `@adobe/fetch` and returns `Response2` in
isolation — verified by a standalone local test. But something in the module
graph after #2822 added `metrics-emf.js` imports to both `RedirectsController`
and `AsoOverlayKeyHandler` now resolves to `Response6` at bundle-time along
one of the code paths.

The exact esbuild module-resolution behavior is opaque; rather than chase
it in the bundle and risk regressing again on any future import graph
change, normalize at the outermost seam so the AWS adapter always sees an
`@adobe/fetch`-compatible Response.

## Fix

New `ensureFetchResponseWrapper` installed as the OUTERMOST wrapper (runs
LAST on the response path, immediately before helix-universal's aws-adapter
serializes the response). On the fast path (Response is already an
`@adobe/fetch` Response), the wrapper is a zero-cost passthrough. On the
slow path (native Response reached us), it rebuilds the response as an
`@adobe/fetch` Response with the same status, headers, and body.

This is defense-in-depth: it fixes the specific interaction from #2822
AND protects against future PRs that could introduce the same shift via
different code paths, without requiring bundle-level bisection every time.

## Why we didn't catch this locally

Two independent gaps that this PR closes:

1. **Unit tests import from source** (`src/controllers/redirects.js`), where
   module resolution goes directly to `node_modules/@adobe/fetch`.
   `createResponse` returns `Response2` — `.raw()` works. Tests pass.
2. **CI's `bundle-build` step** invokes the bundled Lambda but only against
   the `/health` endpoint (JSON, unrelated code path). The overlay path is
   not exercised at bundle time.

The rewrap wrapper closes the runtime gap; a follow-up ticket
(SITES-48140 follow-up thread) tracks adding a bundle-level integration
test that hits `/config/dev/*/redirects.txt` against the bundled lambda so
this class of regression is caught pre-merge.

## Test plan

- [x] 7 unit tests on the new wrapper (fast path, slow path, 200/304/404,
      empty body, non-object results, error propagation)
- [x] Existing suites still green (redirects.test.js + support/ — 4908
      passing across the two directories)
- [x] Lint clean, type-check clean
- [ ] Post-deploy on dev: verify GET /config/dev/cm-p117653-e365853/redirects.txt
      returns 200/304/404 as expected (no more 500)
- [ ] Then unblocks Alina's TTL bump verification and the fleet-wide
      customer verification (18 envs, task 11)

## Non-goals

Not re-bisecting the exact esbuild module-resolution to identify which
transitive import triggered the shift — that's opaque and time-boxed;
the wrapper defends regardless. If desired, follow-up work can bisect
by re-introducing pieces of #2822's imports one at a time under the
new bundle-integration-test net.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@github-actions

Copy link
Copy Markdown

This PR will trigger a patch release when merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant