Skip to content

improv: omit passed items from disk writes to reduce storage usage#773

Merged
younglim merged 5 commits into
masterfrom
improv/omit-passedItems-data
Jul 6, 2026
Merged

improv: omit passed items from disk writes to reduce storage usage#773
younglim merged 5 commits into
masterfrom
improv/omit-passedItems-data

Conversation

@younglim

@younglim younglim commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Problem

On large scans, the Docker container running Oobee failed with two separate errors that compound each other:

Error 1 — Disk exhaustion when writing scanItems.json:

Error: ENOSPC: no space left on device, write

The passed category (elements that passed accessibility checks) dominates the size of scanItems.json on large sites — axe-core records hundreds of passing elements per page across potentially thousands of pages.

Error 2 — Playwright post-close crash in writeSummaryPdf:

Error: Object with guid response@... was not bound in the connection

After writeSummaryPdf closes the headless browser, deferred setImmediate callbacks from Playwright's in-process IPC factory fire attempting to dispatch to an already-disposed Response object. This arrives as an uncaughtException. Additionally, Winston's handleExceptions: true transport registers its own uncaughtException listener that fires before psTreeHandler and calls process.exit(1) by default — so the psTreeHandler suppression never had a chance to run.

Error 3 — /tmp ENOSPC preventing summary.pdf generation:

browserType.launchPersistentContext: ENOSPC: no space left on device, mkdtemp '/tmp/playwright-artifacts-...'

Chrome's --disable-dev-shm-usage flag (required in Docker where /dev/shm is only 64MB) redirects shared memory writes to /tmp. During the crawl phase, Chrome fills /tmp with SHM segments and temp files. When writeSummaryPdf later tries to launch a second browser for PDF rendering, Playwright cannot create its artifacts directory in /tmp.

Why Error 3 was hidden before: smaller scans don't accumulate enough temp data to exhaust /tmp, or large scans previously died at Error 1 before reaching writeSummaryPdf. Once Error 1 is fixed, Error 3 is exposed on large scans.


Changes

src/mergeAxeResults.ts

  • pushResults: Added if (category !== 'passed') guards around both itemsStore.appendPageItems call sites (custom flow and standard flow). Passed item counts and pagesAffected metadata are still tracked in memory; only the intermediate JSONL temp files on disk are skipped.
  • buildHtmlGroups loop: Removed 'passed' from the category array. No passed items are stored in itemsStore anymore, and convertItemsToReferences already excludes htmlGroups for passed — so iterating over it was a no-op.
  • writeSummaryPdf: Flushes TMPDIR before launching the PDF browser, reclaiming space consumed by the crawl phase's temp files.

src/mergeAxeResults/jsonArtifacts.ts

  • writeJsonAndBase64Files: Destructures passed out of items before building the scanItems write payload, so scanItems.json and its compressed scanItems.json.gz.b64 no longer contain passed item details.

src/generateHtmlReport.ts

  • Added a defensive fallback: if scanItems.json does not contain a passed key (files produced after this change), inject an empty-category object before calling convertItemsToReferences to prevent a crash when regenerating reports from existing JSON files.

src/constants/common.ts

  • getClonedProfilesWithRandomToken: In Docker, sets process.env.TMPDIR to <cwd>/Chromium Support/tmp. This redirects all Chrome and Playwright temp file creation away from /tmp into the app's own writable directory. The path is kept short (e.g. /app/oobee/Chromium Support/tmp/...) to stay within Linux's 107-byte Unix socket path limit — longer paths (such as including the full randomToken) cause Chrome to crash with SIGTRAP when it tries to create IPC sockets.

src/combine.ts

  • Extended psTreeHandler to suppress Playwright's was not bound in the connection uncaughtException. This error is a benign post-cleanup artefact — the browser was already closed and the scan output is complete.

src/logs.ts

  • Set exitOnError: false on both consoleLogger and silentLogger. Winston's handleExceptions: true transport registers its own uncaughtException listener that previously fired before psTreeHandler and called process.exit(1), preventing the suppression from ever running. With exitOnError: false, Winston logs the exception and returns, allowing psTreeHandler to decide whether to continue (benign errors) or rethrow (real errors that should terminate the process).

src/utils.ts

  • cleanUp: Removes the TMPDIR directory (Chromium Support/tmp) on exit.

What is preserved

Data Still available?
Passed item count (totalItems) ✅ In memory + scanData.json via allIssues.totalItems
Passed rule names & counts scanIssuesSummary.json
Passed rule/count metadata in HTML report ✅ Embedded scanItems-light payload (built from in-memory allIssues.items.passed)
Per-page passed item details on disk ❌ Removed (was the source of disk exhaustion)

Reverting

All changed lines in the disk-space optimization are annotated with // Disk space: and // To revert: comments in-code. To restore the original behaviour:

  1. mergeAxeResults.tspushResults: Remove the two if (category !== 'passed') guards; the original unconditional appendPageItems calls are shown in the adjacent comments.
  2. mergeAxeResults.tsbuildHtmlGroups loop: Restore 'passed' to the category array: ['mustFix', 'goodToFix', 'needsReview', 'passed'].
  3. jsonArtifacts.ts: Remove the const { passed: _passedItems, ...itemsWithoutPassed } = items; line and restore the original argument: { oobeeAppVersion: allIssues.oobeeAppVersion, ...items }.
  4. generateHtmlReport.ts: Remove the if (!scanItemsAll.passed) { ... } fallback block.
  5. combine.ts: Remove the if (err.message?.includes('was not bound in the connection')) suppression block from psTreeHandler.
  6. logs.ts: Remove exitOnError: false from both logger configurations. Note: this must be reverted together with step 5 — exitOnError: false exists solely to prevent Winston from calling process.exit(1) before psTreeHandler can suppress the Playwright error. Removing it without also removing the step 5 suppression would cause Winston to exit the process before psTreeHandler runs again.
  7. common.tsgetClonedProfilesWithRandomToken: Remove the if (fs.existsSync('/.dockerenv')) block that sets TMPDIR.
  8. mergeAxeResults.tswriteSummaryPdf: Remove the if (process.env.TMPDIR) { fs.emptyDirSync(...) } block.
  9. utils.tscleanUp: Remove the if (process.env.TMPDIR) removal block.

younglim added 3 commits July 2, 2026 17:09
- mergeAxeResults.ts: skip itemsStore.appendPageItems for 'passed' category
  in pushResults (both custom flow and standard flow paths)
- mergeAxeResults.ts: remove 'passed' from buildHtmlGroups loop (no items
  stored in itemsStore for passed; convertItemsToReferences already excludes
  htmlGroups for passed)
- jsonArtifacts.ts: exclude passed from scanItems.json write payload;
  passed counts remain available via scanData.json and the embedded
  scanItems-light payload in the HTML report
- generateHtmlReport.ts: add fallback empty-category for passed when
  regenerating a report from scanItems.json that lacks the passed key

All changes annotated with revert instructions.
After writeSummaryPdf closes the browser, a deferred setImmediate callback
fires from Playwright's in-process IPC factory attempting to dispatch a
message to an already-disposed Response object. This arrives as an
uncaughtException through psTreeHandler and crashes the process.

Suppress 'was not bound in the connection' errors the same way the
Crawlee ps-tree locale error is suppressed — they are benign post-cleanup
artifacts and do not affect scan output.
Winston's handleExceptions: true registers its own uncaughtException listener
that fires before psTreeHandler and calls process.exit(1) by default.
This caused the Playwright post-close 'not bound in the connection' error
to exit the process before psTreeHandler could suppress it.

Setting exitOnError: false on both consoleLogger and silentLogger makes
Winston log the exception without exiting, allowing psTreeHandler in
combine.ts to decide whether to continue (benign errors) or rethrow
(real errors that should terminate the process).
@younglim younglim requested a review from WilsonkwSheng July 2, 2026 09:52
@WilsonkwSheng

WilsonkwSheng commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

LGTM

younglim and others added 2 commits July 3, 2026 15:57
…ENOSPC in Docker

In constrained containers, Chrome's --disable-dev-shm-usage writes shared memory
to /tmp, exhausting it before PDF generation can launch a second browser. This
redirects TMPDIR into the scan's existing Chromium Support directory (scoped by
randomToken), flushes it before PDF browser launch, and cleans it up on exit.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The previous path included the full randomToken (e.g. oobee-20260703_..._997/tmp),
which when combined with Playwright's subdirectory exceeded Linux's 107-byte Unix
socket path limit, causing Chrome to crash with SIGTRAP on launch.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@younglim younglim merged commit 3017147 into master Jul 6, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants