fix(cwv): only exclude genuinely-gone (404/410) URLs, not bot-blocked ones

[habansal]

What & why

Implements SITES-47218 (part of the datacom.com investigation, SITES-47216).

CWV candidates come from RUM field data — real users successfully load these pages — so a request our infrastructure can't complete (a 403 bot-block, 429, 5xx, or timeout) does not mean the page is gone. The previous isUrl4xxOrFailed dropped any URL returning 4xx or failing the HEAD check, so a site that bot-blocks our crawler (e.g. datacom.com behind Cloudflare) lost all of its valid CWV opportunities — an empty result that then reads to the customer as "resolved".

Change

• isUrl4xxOrFailed → isUrlGone: only 404/410 are treated as "gone". 403/401/429, 5xx, and transient/timeout failures now keep the URL.
• Preserves the original 404/clientlib/sling filtering intent (SITES-40803) without the over-suppression.

Why not switch the client to @adobe/fetch?

That would "pass" datacom today, but it's a site-specific result (N=1) and fragile (useless vs stricter configs / Akamai/Imperva). Not gating RUM CWV on reachability is the robust fix.

Test plan

• Unit tests: isUrlGone (404/410 → gone; 403/200/500/throw → not gone); buildCWVAuditResult retains a 403 URL (regression for SITES-47218); group entries untouched. 9/9 pass locally.
• Lint clean.
• CI green (coverage on Node 24).

Real-data validation (dev, 2026-07-02)

Deployed to an isolated dev Lambda alias (spacecat-services--audit-worker:habansal-cwv, ExecutedVersion 8343) and invoked directly against real datacom.com in the dev VPC (egress Cloudflare-403'd, same as prod):

• Step 1 (buildCWVAuditResult): Total=112, Reported=16, 0 excluded — old code logged Excluded 16 (4xx or HEAD failed) → cwv: [].
• Step 2: created 3 new CWV suggestions.
• Same site / same 403 egress / same RUM data — only the code changed; result flipped 0 → 16 and restored the previously-missing suggestions.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[MysticatBot]

Hey @habansal,

Verdict: Approve - well-scoped bug fix with clear rationale, good test coverage, and a thoughtful safety guard.
Complexity: LOW - small diff, single module, no risk signals.
Changes: Narrows URL exclusion logic from "any 4xx or failure" to "only 404/410" and adds a safety guard that retains all URLs when every candidate reports gone (2 files).

Non-blocking (2): minor issues and suggestions

• suggestion: The allGone guard requires urlEntries.length > 1, so a site with exactly 1 URL entry that returns 404 is still excluded (correct behavior), but this asymmetry has no test documenting the intent. A test like it('excludes a single URL that is genuinely gone (guard does not apply)') would prevent accidental regressions if the guard condition is refactored. - src/cwv/cwv-audit-result.js:157
• nit: The integration-level buildCWVAuditResult tests verify exclusion with 404 but not 410. Since isUrlGone unit tests cover 410, this is low-risk, but an integration case would confirm end-to-end wiring for both codes. - test/audits/cwv/cwv-audit-result.test.js:111

---

_{Skill: pr-review | Model: us.anthropic.claude-opus-4-6-v1[1m] | Duration: 1m 17s | Cost: $3.64 | Commit: 1d65a8f0a85db668a2e10ceb67fffcf6ed7274d9
If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[github-actions]

This PR will trigger a patch release when merged.

[MysticatBot]

Hey @habansal,

⚠ Degraded review - no spec document was found for this change (searched the PR links, the touched repos' docs, the architecture/guidelines docs, and linked Jira). This review covers code-level quality but could not validate the change against an agreed design, so confidence is reduced. Add a spec link (PR template section 4) and re-request review for a full-confidence pass.

Verdict: Approve - clean bug fix, well-tested, correct semantics.
Complexity: LOW - small diff, single module, no risk signals.
Changes: Narrows URL exclusion from "any 4xx or failure" to "only 404/410 (genuinely gone)" and removes the all-gone safety guard that is no longer needed under the tightened semantics (2 files).

Non-blocking (1): minor issues and suggestions

• nit: PR description still advertises "Site-wide safety guard: if every candidate URL reports gone, keep them all and warn" but that code was added in commit 1 and removed in commit 2. A reader arriving at this PR later will be confused about what actually shipped. Consider updating the body to reflect the final state.

Previously flagged, now resolved

• Safety guard removal justified by tightened isUrlGone semantics (commit 2 cleanly removes it with rationale in message).

---

_{Skill: pr-review | Model: us.anthropic.claude-opus-4-6-v1[1m] | Duration: 1m 28s | Cost: $2.97 | Commit: 132ed190200956d708f71f773a9baf0521d49928
If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[habansal]

✅ Real-data validation on dev (2026-07-02)

Validated end-to-end against real datacom.com traffic in the dev VPC (whose egress Cloudflare 403s exactly as prod).

Method: deployed this branch to an isolated dev Lambda alias (spacecat-services--audit-worker:habansal-cwv, ExecutedVersion 8343) and invoked it directly for siteId 3fd44cf2-c9f8-40ee-9d9b-22c296ebb633 — so the shared latest alias / other PRs couldn't interfere.

Step 1 — buildCWVAuditResult (this fix):

[audit-worker-cwv] ... baseURL: https://datacom.com | Total=112, Reported=16 | Homepage: included | Top15 pages: 15 | Pages above threshold: 0

0 excluded (no Excluded line). Old code logged Excluded 16 URL(s) (4xx or HEAD failed) → cwv: []. All 16 URLs retained despite the 403s.

Step 2 — syncOpportunitiesAndSuggestions: 15 of 16 CWV entries have failing metrics → created 3 new CWV suggestions.

Same site / same 403-blocking egress / same RUM data — only the code changed; result flipped 0 → 16 and produced the previously-missing suggestions.

Orthogonal note: the dev code-import step errored (Cannot read properties of null (reading 'owner')) — a dev-only data gap (no repo owner), unrelated to CWV; didn't block suggestions.