feat(prerender): refactoring prerender audit by anuj-adobe · Pull Request #2488 · adobe/spacecat-audit-worker

anuj-adobe · 2026-05-07T06:22:53Z

Summary

Addresses prerender scrape failures and budget waste through four targeted changes:

Fix 1 — SQL 404 filter: Add AND status = 200 to top-agentic-urls-by-limit.sql — eliminates ~38% of failures by never queuing URLs the CDN already reports as 404
Fix 2 — Permanent 410 exclusion: Mark 410 Gone URLs with gone: true in status.json; excluded from all future daily batches
Fix 3 — Proactive bot-block detection: Call detectBotBlocker() before fetching any candidate URLs; skip the entire batch when confidence ≥ 0.95 — addresses ~10% of failures
Fix 4 — Skip already-deployed URLs: Filter URLs whose suggestions already have edgeDeployed or coveredByDomainWide (when the domain-wide rule is still active) out of the daily scrape batch — re-scraping these adds no value since they already score 100% in the UI

Fix 5 (null-guard for shadow DOM TypeError) is in a separate content-scraper PR. Fix 6 (429 Retry-After) is deferred.

Refactoring

handler.js decomposed: submitForScraping split into buildScrapeResult, buildSlackBatch, buildDailyBatch; handleAiOnlyMode moved to ai-only-handler.js
opportunity-utils.js: findPrerenderOpportunity() and findDeployedDomainWideSuggestion() extracted as shared helpers, reused across getDomainWideSuggestionDeployedAtEdge, detectWrongEdgeDeployedStatus, and the new getDeployedOrCoveredPathnames()
url-selector.js: new getDeployedOrCoveredPathnames() helper; getTopOrganicUrlsFromSeo, getTopAgenticUrls, getRecentlyProcessedPathnames, isNotRecentUrl consolidated here
status-json.js: readSiteStatusJson + uploadStatusSummaryToS3 extracted from handler

Test plan

All prerender tests pass (330 tests, 100% coverage on modified files)
Fix 1: top-agentic-urls-by-limit.sql only returns status=200 URLs
Fix 2: 410 response sets gone: true; URL excluded from future batches
Fix 3: detectBotBlocker confidence ≥ 0.95 skips the entire batch
Fix 4: URLs with edgeDeployed or coveredByDomainWide filtered out of daily batch; rollback of domain-wide rule auto-restores those URLs to the queue

References

LLMO-4040 — Scrape failure analysis
Wiki: Reduce prerender content scraping budget

🤖 Generated with Claude Code

…uit breaker, 403 domain block, and 429 rate limit - Fix 1: SQL filter `AND status = 200` in top-agentic-urls query prevents ~38% of failures (404 URLs never queued) - Fix 2: Per-URL `circuitBreakerOpen: true` for 410 Gone responses (permanent, never re-queued) - Fix 3: Domain-level 403 circuit breaker in status.json root; threshold MAX(50, 50% of batch), 2d/4d/8d progressive backoff, auto-restores when 403 count drops below threshold - Fix 5: Domain-level `rateLimitedUntil` in status.json root for 429 responses; honours Retry-After header, auto-expires Extracts readSiteStatusJson() and computeDomainBlock() helpers to avoid S3 read duplication and improve testability. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…modules Move new helpers introduced in the preceding fix commit to focused modules: - utils/domain-block.js: computeDomainBlock() pure function (403 circuit breaker logic) - utils/status-json.js: readSiteStatusJson() and uploadStatusSummaryToS3() (all S3 status.json I/O) - utils/utils.js: normalizePathname() export (was private in handler.js, needed by status-json) handler.js is now a thin orchestrator: imports and calls these helpers rather than defining them. uploadStatusSummaryToS3 is re-exported from handler.js for backward compat with existing tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-05-07T06:33:32Z

This PR will trigger a minor release when merged.

…r proactive check Fix 3 no longer needs state in status.json. Instead of accumulating 403 counts across runs and writing domainBlock/skipUntil/consecutiveBlocks, we call detectBotBlocker() once before submitting the scrape batch. If the domain is actively blocking crawlers (confidence >= 0.95), we skip the batch immediately — no backoff intervals, no stored state to manage. Fix 5 (rateLimitedUntil for 429) is kept: a single HEAD probe cannot predict rate limiting triggered by batch volume, so the reactive Retry-After window remains. Removes utils/domain-block.js entirely. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Move functions from handler.js into dedicated modules by concern: - utils/opportunity-utils.js: suggestion/opportunity management (9 functions) - utils/url-selector.js: URL candidate assembly (5 functions) - utils/scrape-utils.js: S3 content access and scrape job helpers (7 functions) - utils/citability.js: page citability record writes (1 function) - ai-only-handler.js: Slack-triggered reconciliation mode (1 function) - sendPrerenderGuidanceRequestToMystique moved into guidance-handler.js handler.js is now a thin orchestrator of exported step functions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…mports to source modules Re-exports like `export { X } from './utils/y.js'` in handler.js were a backward-compat shim for tests. Tests now import each function directly from its owning module. handler.js no longer re-exports anything it does not define. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…gned in follow-up PR Rate limiting behaviour varies too much (seconds vs hours) to handle reliably with a fixed Retry-After default. Removing the partial implementation to keep this PR clean; proper 429 handling will be addressed in a dedicated follow-up. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…=200 URLs Reverts the status=200 change from the shared SQL to avoid impacting other audits. Adds a new top-agentic-live-urls-by-limit.sql and a parallel getTopAgenticLiveUrlsFromAthena function (reusable by any audit) that restricts results to successfully-served URLs only. Prerender switches to this variant to avoid submitting 404/410 URLs for scraping (Fix 1). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… add live variant Keeps the existing function body untouched to minimise diff and regression risk. Adds getTopAgenticLiveUrlsFromAthena as a parallel new function. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…add live variant - Adds optional statuses[] param to getTopAgenticUrlsFromAthena; when non-empty routes to new top-agentic-urls-by-status-and-limit.sql to avoid touching the original SQL and prevent any regression for existing callers - Adds getTopAgenticLiveUrlsFromAthena as a one-liner wrapper passing [200] - Prerender switches to getTopAgenticLiveUrlsFromAthena to skip 404/410 URLs (Fix 1) - Unit tests cover status filter in SQL output and the live URL function Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…omAthena Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ailyBatch from submitForScraping Eliminates 7 pre-declared let variables and the large nested else block. submitForScraping now reads as a clear sequence of three distinct paths (ai-only, CSV explicit, Slack/daily) with no behavior changes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…bmitForScraping Checks detectBotBlocker before the expensive getTopOrganicUrlsFromSeo and getIncludedURLs calls. Skipped for Slack-triggered runs (manual override). Also removes the batch.domainBlocked sentinel from buildDailyBatch. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…y scrape batch URLs whose suggestions already have edgeDeployed or coveredByDomainWide (when the domain-wide rule is still active at the CDN edge) gain nothing from re-scraping. Filtering them out before building the daily batch reduces wasted scrape budget. - Add getDeployedOrCoveredPathnames() in url-selector.js, reusing shared findPrerenderOpportunity/findDeployedDomainWideSuggestion helpers from opportunity-utils.js; coveredByDomainWide is only excluded when the domain-wide suggestion still has edgeDeployed, so rollback auto-restores those URLs to the scrape queue on the next audit run - Refactor opportunity-utils.js: extract findPrerenderOpportunity() and findDeployedDomainWideSuggestion() as reusable exports; remove redundant typeof guard from getDomainWideSuggestionDeployedAtEdge - Fix ai-only-mode.test.js import: handleAiOnlyMode moved to ai-only-handler.js - Remove Fix N labels from inline comments Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…on, scrape-utils - handler.js:225-227: add domainBlocked test (detectBotBlocker confidence >= 0.95) - status-json.js:109: add wasGone test (prior entry had gone=true, current is non-410) - scrape-utils.js:79-81: add getScrapedHtmlFromS3 direct test with all S3 fetches rejected - query-builder.js: remove unreachable guard in buildStatusFilter (caller already guards) - submitForScraping describe: extend timeout to 10000ms (detectBotBlocker makes real HTTP call) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

TierClient.createForSite was added in processContentAndGenerateOpportunities without corresponding test mocks, causing real HTTP calls that hang under Mocha's default timeout. Added isPaidLLMOCustomer stub in the esmock 3rd arg (global mocks) for all 14 affected esmock calls in handler.test.js. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

utils.js (isPaidLLMOCustomer) must stay in the 2nd arg (childmocks) so it intercepts handler.js's direct import and prevents CI timeouts from TierClient HTTP calls. s3-utils.js, html-comparator.js, agentic-urls.js, and opportunity.js are transitive imports (or require global scope) and must stay in the 3rd arg (globalmocks) so they are intercepted throughout the isolated module tree. Mixing direct-import mocks into the 3rd arg alongside global ones breaks esmock's isolation — keep only utils.js in the 2nd arg and everything else in the 3rd arg. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The 'should exclude URL with needsPrerender=true and isDeployedAtEdge=true' test was introduced without mocking utils.js, leaving isPaidLLMOCustomer to make a real TierClient HTTP call that hangs in CI. That call is now properly mocked via utils.js in the 2nd esmock arg (childmocks), but the test still takes ~1.2s locally due to esmock module-tree loading overhead — which exceeds the 2000ms global timeout on slower CI runners. Set this.timeout(5000) consistent with other heavy esmock tests in this file. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Eight tests that esmock handler.js with multiple mocks take ~1–1.25s locally due to module-tree loading overhead, which exceeds the 2000ms global timeout on CI runners that are ~2x slower. Add this.timeout(5000) to each affected test, consistent with the pattern already used by other heavy esmock tests in this file. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

codecov · 2026-05-12T12:20:24Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

… statuses parameter Replace the two separate SQL files (top-agentic-urls-by-limit and top-agentic-live-urls-by-limit) with a single top-agentic-urls-by-status-and-limit SQL that accepts a {{statusFilter}} placeholder, aligning with the approach in the companion refactoring PR (#2488). getTopAgenticLiveUrlsFromAthena now delegates to getTopAgenticUrlsFromAthena with statuses=[200]. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…detection, skip deployed URLs (#2510) ## Problem Error-category analysis from [LLMO-4040](https://jira.corp.adobe.com/browse/LLMO-4040) identified **311,086 scrape failures** across a representative window. The breakdown by root cause: | Error Category | Failures | % of Total | Root Cause | |---|---|---|---| | HTTP 404 (Not Found) | 117,462 | **37.8%** | Agentic URL source includes URLs CDN already serves as 404; scraper queues them anyway | | HTTP 403 (Forbidden / Bot Block) | 32,333 | **10.4%** | Domains actively blocking the scraper; no per-domain detection before submitting | | HTTP 429 (Too Many Requests) | ~3,284 URLs / 64 domains | ~1.1% | Scraper retries immediately without respecting `Retry-After` header | | HTTP 410 (Gone) | Small | <1% | Permanently removed pages re-queued indefinitely | | TypeError (shadow DOM null) | 611 URLs / 103 domains | ~0.2% | Null-reference in shadow DOM expansion mid-evaluation | | Already-deployed URLs re-scraped | Hundreds/run per deployed domain | Ongoing | URLs already deployed to the CDN submitted to scraper despite scoring 100% — for sites with stale `status.json` error entries (e.g. ricekrispies.ca: 351 of 417 covered URLs had `scrapingStatus: error`) this triggered fresh failures on every daily run | **Total addressable by fixes 1, 3, and 4: ~48%+ of failures + ongoing per-run budget waste.** Full analysis and fix proposals: [Reduce prerender content scraping budget](https://wiki.corp.adobe.com/spaces/AEMSites/pages/3871795321/Reduce+prerender+content+scraping+budget) --- ## Summary This PR implements fixes 1, 3, and 4 in `spacecat-audit-worker`: - **Fix 1 — SQL 404 filter**: Switch Athena query to `getTopAgenticLiveUrlsFromAthena` (new `top-agentic-live-urls-by-limit.sql` with `AND status = 200`) — eliminates 404 failures at source. **~117,000 failures (~38%)** - **Fix 2 — Permanent 410 exclusion**: Covered indirectly by Fix 1 — `getTopAgenticLiveUrlsFromAthena` only returns URLs with `status=200` CDN rows. A 410 Gone URL has no 200 rows in the CDN data, so it never enters the candidate pool and is never submitted for scraping. - **Fix 3 — Bot-block detection with sticky skip**: Two-stage detection to avoid re-scraping bot-blocked domains. (1) **Pre-scrape sticky skip**: `submitForScraping` reads `scrapeForbidden` from `status.json`; if a prior run confirmed blocking within the last 3 days, the entire batch is skipped immediately (`domainBlocked: true`) — no URLs submitted, no wasted scrape budget. Slack-triggered runs bypass this so operators can force a re-scrape. (2) **Post-scrape reactive detection**: after results arrive, if the 403 ratio exceeds the threshold, `detectBotBlocker()` from `@adobe/spacecat-shared-utils` is called; if it returns a named CDN blocker (Cloudflare, Imperva, Akamai, Fastly, CloudFront) at confidence ≥ 0.99, `scrapeForbidden: true` is written to `status.json` so the next daily run's sticky check fires. Threshold 0.99 avoids false positives from HTTP/2 stream errors, which produce the same 403 codes as real bot blocking. **~32,000 failures (~10%)** - **Fix 4 — Skip already-deployed URLs**: Filter URLs where `isDeployedAtEdge` is set in `status.json` out of the daily scrape batch. This flag is written per-URL by the content scraper and naturally captures both directly-deployed URLs and domain-wide CDN coverage. Re-scraping these adds no value — they already score 100% in the UI, and for sites with stale error entries in `status.json` they cause fresh failures on every run. Fix 5 (TypeError null-guard for shadow DOM) is in a separate `spacecat-content-scraper` PR. Fix 6 (429 Retry-After) is deferred. ## Context This PR is a cherry-picked, non-refactoring extraction from #2488, which bundled the same fixes together with a larger `handler.js` decomposition (splitting into `url-selector.js`, `opportunity-utils.js`, `status-json.js`, `ai-only-handler.js`). Splitting allows the fixes to land and be validated independently before the structural refactor follows. **Follow-up:** #2488 will be rebased on top of this PR once it merges and serves as the refactoring PR (PR2). ## Test plan - [ ] All tests pass (`handler.test.js`, `ai-only-mode.test.js`, `query-builder.test.js`) - [ ] Fix 1: `top-agentic-live-urls-by-limit.sql` only returns status=200 URLs - [ ] Fix 3: `status.json` with `scrapeForbidden: true` set within 3 days skips the batch (`domainBlocked: true`) in `submitForScraping`; post-scrape high 403 ratio triggers `detectBotBlocker` at 0.99 confidence and writes `scrapeForbidden` to `status.json` - [ ] Fix 4: URLs with `isDeployedAtEdge: true` in `status.json` are filtered out of the daily batch - [ ] `buildStatusFilter` rejects non-integer and out-of-range status codes (SQL injection prevention) - [ ] esmock-heavy tests have `this.timeout(5000)` to prevent CI timeouts on slower runners - [ ] No `url-selector.js`, `status-json.js`, or `ai-only-handler.js` introduced — all changes in existing `handler.js` ## References - [LLMO-4040](https://jira.corp.adobe.com/browse/LLMO-4040) — Scrape failure error-category analysis - [Wiki: Reduce prerender content scraping budget](https://wiki.corp.adobe.com/spaces/AEMSites/pages/3871795321/Reduce+prerender+content+scraping+budget) - #2488 — Original PR (fixes + refactoring); becomes PR2 after this merges 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: mahAnuj <mahajananuj07@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>

anuj-adobe and others added 2 commits May 7, 2026 11:52

anuj-adobe and others added 12 commits May 7, 2026 12:30

refactor(prerender): rename circuitBreakerOpen to gone for 410 URLs

836ddf6

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(prerender): remove duplicate JSDoc comment on getTopAgenticUrlsFr…

12db876

…omAthena Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

anuj-adobe changed the title ~~fix(prerender): reduce scrape error rate (404 filter, 410 circuit breaker, 403 domain block, 429 rate limit)~~ feat(prerender): reduce scrape budget — SQL 404 filter, 410 exclusion, bot-block detection, skip deployed URLs May 7, 2026

anuj-adobe marked this pull request as ready for review May 7, 2026 14:17

anuj-adobe and others added 5 commits May 12, 2026 13:30

anuj-adobe temporarily deployed to dev-branches May 12, 2026 12:19 — with GitHub Actions Inactive

anuj-adobe mentioned this pull request May 12, 2026

fix(prerender): reduce scrape error rate — SQL 404 filter, bot-block detection, skip deployed URLs #2510

Merged

7 tasks

anuj-adobe changed the title ~~feat(prerender): reduce scrape budget — SQL 404 filter, 410 exclusion, bot-block detection, skip deployed URLs~~ feat(prerender): refactoring prerender audit May 18, 2026

anuj-adobe marked this pull request as draft May 18, 2026 07:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(prerender): refactoring prerender audit#2488

feat(prerender): refactoring prerender audit#2488
anuj-adobe wants to merge 19 commits into
mainfrom
feat/LLMO-prerender-scrape-error-reduction

anuj-adobe commented May 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 7, 2026 •

edited

Loading

Uh oh!

codecov Bot commented May 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

anuj-adobe commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Refactoring

Test plan

References

Uh oh!

github-actions Bot commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented May 12, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

anuj-adobe commented May 7, 2026 •

edited

Loading

github-actions Bot commented May 7, 2026 •

edited

Loading