Skip to content

revert(aso-overlay): rollback #2822 to fix dev 500 (response.headers.raw is not a function)#2838

Closed
abhishekgarg18 wants to merge 1 commit into
mainfrom
fix/revert-2822-emf-500-regression
Closed

revert(aso-overlay): rollback #2822 to fix dev 500 (response.headers.raw is not a function)#2838
abhishekgarg18 wants to merge 1 commit into
mainfrom
fix/revert-2822-emf-500-regression

Conversation

@abhishekgarg18

Copy link
Copy Markdown
Member

Summary

Reverts #2822 (`feat(aso-overlay): CloudWatch EMF instrumentation for READ path`) to restore the dev overlay endpoint. Since #2822 deployed to dev (2026-07-16T03:52Z), every GET /config/dev/*/redirects.txt returns HTTP 500 with:

```
x-error: response.headers.raw is not a function
```

Verified reproduction on dev with a valid ASO_OVERLAY_API_KEY, on multiple tenants:

  • `cm-p117653-e365853` (Maksym's test tenant)
  • `cm-p128729-e2179647` (Okta dev)

Consistent, non-intermittent, on both the 200 and 304 conditional-GET code paths.

Timeline

Time (UTC) Event
2026-07-15T20:23Z Maksym verified end-to-end key rotation working on dev cm-p117653-e365853 (no `invalid X-ASO-API-Key` errors)
2026-07-16T00:03Z Maksym verified full purge-on-Deploy chain: Deploy → S3 write → Mystique purge (`fastly_purge OK, 24ms`) → edge flipped in ~2s
2026-07-16T03:52Z #2822 merged (v1.655.0 promoted #2822)
2026-07-16T~10:30Z 500s observed on dev overlay endpoint

Nothing else in the overlay hot path changed in this window. Other PRs merged after this window (#2820, #2833, #2804, #2823) don't touch redirects.js, aso-overlay-key-handler.js, or the metrics-emf infrastructure.

Root cause (in progress)

The error `response.headers.raw is not a function` is thrown at `@adobe/helix-universal/aws-adapter.js:254`:

```js
...splitHeaders(response.headers.raw(), con.log),
```

That helper expects `@adobe/fetch`-style Headers (which have `.raw()`). The Response returned to aws-adapter has a native Web-Fetch-API Headers instance (no `.raw()`).

Local repro confirms `createResponse` from `@adobe/spacecat-shared-http-utils` correctly returns `@adobe/fetch` Response with `.raw()` working (see snippet below). So the failure is Lambda-bundle-specific — likely an esbuild resolution/tree-shaking interaction introduced by #2822's new imports (`aso-overlay-metrics.js` constants + widened use of `metrics-emf.js`).

```js
// Local test (from within the repo)
import { createResponse } from '@adobe/spacecat-shared-http-utils';
const r = createResponse('body', 200, { 'content-type': 'text/plain' });
console.log(typeof r.headers.raw === 'function'); // true — works locally
```

Not fully root-caused yet. Reverting to unblock dev while investigation continues. This is the safest immediate fix — every branch of the overlay endpoint is failing, and the metrics that #2822 added are non-critical (dashboards can wait a day).

Follow-up plan (post-revert)

  1. Bring feat(aso-overlay): CloudWatch EMF instrumentation for READ path (SITES-48140) #2822's changes back behind a functional-parity test that exercises the actual helix-deploy bundle (not just source-level tests) — mirroring the SITES-45260 pattern documented in CLAUDE.md ("works in tests, breaks in production is the bundling layer")
  2. Bisect: try re-adding `aso-overlay-metrics.js` alone, then `metrics-emf.js` widening, to identify the exact bundling interaction
  3. Consider whether all `emitMetric` call sites should be wrapped in an isolate-and-log outer try/catch at the wrapper level (defense-in-depth beyond the current in-function try/catch)

Impact if not reverted

  • All ASO dispatcher-overlay traffic on dev returns 500 → no dispatcher can fetch overlays → broken-backlinks redirect layer non-functional on dev
  • Blocks Alina's TTL bump verification (task 37)
  • Blocks any stage/prod promotion (dev is the release gate)

Test plan

  • Local unit tests still pass after revert (18/18 in redirects.test.js, 8/8 in aso-overlay-key-handler.test.js)
  • After merge + deploy: verify dev endpoint returns 200/304/404 as expected with valid key
  • Confirm Maksym's earlier end-to-end verification still passes

🤖 Generated with Claude Code

@abhishekgarg18 abhishekgarg18 added the bug Something isn't working label Jul 16, 2026
@codecov

codecov Bot commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@abhishekgarg18

Copy link
Copy Markdown
Member Author

Superseded by #2839 — going with the forward-fix (rewrap wrapper) instead of a revert so #2822's EMF instrumentation stays live once the fix deploys.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant