improv: omit passed items from disk writes to reduce storage usage#773
Merged
Conversation
- mergeAxeResults.ts: skip itemsStore.appendPageItems for 'passed' category in pushResults (both custom flow and standard flow paths) - mergeAxeResults.ts: remove 'passed' from buildHtmlGroups loop (no items stored in itemsStore for passed; convertItemsToReferences already excludes htmlGroups for passed) - jsonArtifacts.ts: exclude passed from scanItems.json write payload; passed counts remain available via scanData.json and the embedded scanItems-light payload in the HTML report - generateHtmlReport.ts: add fallback empty-category for passed when regenerating a report from scanItems.json that lacks the passed key All changes annotated with revert instructions.
After writeSummaryPdf closes the browser, a deferred setImmediate callback fires from Playwright's in-process IPC factory attempting to dispatch a message to an already-disposed Response object. This arrives as an uncaughtException through psTreeHandler and crashes the process. Suppress 'was not bound in the connection' errors the same way the Crawlee ps-tree locale error is suppressed — they are benign post-cleanup artifacts and do not affect scan output.
Winston's handleExceptions: true registers its own uncaughtException listener that fires before psTreeHandler and calls process.exit(1) by default. This caused the Playwright post-close 'not bound in the connection' error to exit the process before psTreeHandler could suppress it. Setting exitOnError: false on both consoleLogger and silentLogger makes Winston log the exception without exiting, allowing psTreeHandler in combine.ts to decide whether to continue (benign errors) or rethrow (real errors that should terminate the process).
Collaborator
|
LGTM |
…ENOSPC in Docker In constrained containers, Chrome's --disable-dev-shm-usage writes shared memory to /tmp, exhausting it before PDF generation can launch a second browser. This redirects TMPDIR into the scan's existing Chromium Support directory (scoped by randomToken), flushes it before PDF browser launch, and cleans it up on exit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The previous path included the full randomToken (e.g. oobee-20260703_..._997/tmp), which when combined with Playwright's subdirectory exceeded Linux's 107-byte Unix socket path limit, causing Chrome to crash with SIGTRAP on launch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On large scans, the Docker container running Oobee failed with two separate errors that compound each other:
Error 1 — Disk exhaustion when writing
scanItems.json:The
passedcategory (elements that passed accessibility checks) dominates the size ofscanItems.jsonon large sites — axe-core records hundreds of passing elements per page across potentially thousands of pages.Error 2 — Playwright post-close crash in
writeSummaryPdf:After
writeSummaryPdfcloses the headless browser, deferredsetImmediatecallbacks from Playwright's in-process IPC factory fire attempting to dispatch to an already-disposedResponseobject. This arrives as anuncaughtException. Additionally, Winston'shandleExceptions: truetransport registers its ownuncaughtExceptionlistener that fires beforepsTreeHandlerand callsprocess.exit(1)by default — so thepsTreeHandlersuppression never had a chance to run.Error 3 —
/tmpENOSPC preventingsummary.pdfgeneration:Chrome's
--disable-dev-shm-usageflag (required in Docker where/dev/shmis only 64MB) redirects shared memory writes to/tmp. During the crawl phase, Chrome fills/tmpwith SHM segments and temp files. WhenwriteSummaryPdflater tries to launch a second browser for PDF rendering, Playwright cannot create its artifacts directory in/tmp.Why Error 3 was hidden before: smaller scans don't accumulate enough temp data to exhaust
/tmp, or large scans previously died at Error 1 before reachingwriteSummaryPdf. Once Error 1 is fixed, Error 3 is exposed on large scans.Changes
src/mergeAxeResults.tspushResults: Addedif (category !== 'passed')guards around bothitemsStore.appendPageItemscall sites (custom flow and standard flow). Passed item counts andpagesAffectedmetadata are still tracked in memory; only the intermediate JSONL temp files on disk are skipped.buildHtmlGroupsloop: Removed'passed'from the category array. Nopasseditems are stored initemsStoreanymore, andconvertItemsToReferencesalready excludeshtmlGroupsforpassed— so iterating over it was a no-op.writeSummaryPdf: FlushesTMPDIRbefore launching the PDF browser, reclaiming space consumed by the crawl phase's temp files.src/mergeAxeResults/jsonArtifacts.tswriteJsonAndBase64Files: Destructurespassedout ofitemsbefore building thescanItemswrite payload, soscanItems.jsonand its compressedscanItems.json.gz.b64no longer containpasseditem details.src/generateHtmlReport.tsscanItems.jsondoes not contain apassedkey (files produced after this change), inject an empty-category object before callingconvertItemsToReferencesto prevent a crash when regenerating reports from existing JSON files.src/constants/common.tsgetClonedProfilesWithRandomToken: In Docker, setsprocess.env.TMPDIRto<cwd>/Chromium Support/tmp. This redirects all Chrome and Playwright temp file creation away from/tmpinto the app's own writable directory. The path is kept short (e.g./app/oobee/Chromium Support/tmp/...) to stay within Linux's 107-byte Unix socket path limit — longer paths (such as including the fullrandomToken) cause Chrome to crash withSIGTRAPwhen it tries to create IPC sockets.src/combine.tspsTreeHandlerto suppress Playwright'swas not bound in the connectionuncaughtException. This error is a benign post-cleanup artefact — the browser was already closed and the scan output is complete.src/logs.tsexitOnError: falseon bothconsoleLoggerandsilentLogger. Winston'shandleExceptions: truetransport registers its ownuncaughtExceptionlistener that previously fired beforepsTreeHandlerand calledprocess.exit(1), preventing the suppression from ever running. WithexitOnError: false, Winston logs the exception and returns, allowingpsTreeHandlerto decide whether to continue (benign errors) or rethrow (real errors that should terminate the process).src/utils.tscleanUp: Removes theTMPDIRdirectory (Chromium Support/tmp) on exit.What is preserved
totalItems)scanData.jsonviaallIssues.totalItemsscanIssuesSummary.jsonscanItems-lightpayload (built from in-memoryallIssues.items.passed)Reverting
All changed lines in the disk-space optimization are annotated with
// Disk space:and// To revert:comments in-code. To restore the original behaviour:mergeAxeResults.ts—pushResults: Remove the twoif (category !== 'passed')guards; the original unconditionalappendPageItemscalls are shown in the adjacent comments.mergeAxeResults.ts—buildHtmlGroupsloop: Restore'passed'to the category array:['mustFix', 'goodToFix', 'needsReview', 'passed'].jsonArtifacts.ts: Remove theconst { passed: _passedItems, ...itemsWithoutPassed } = items;line and restore the original argument:{ oobeeAppVersion: allIssues.oobeeAppVersion, ...items }.generateHtmlReport.ts: Remove theif (!scanItemsAll.passed) { ... }fallback block.combine.ts: Remove theif (err.message?.includes('was not bound in the connection'))suppression block frompsTreeHandler.logs.ts: RemoveexitOnError: falsefrom both logger configurations. Note: this must be reverted together with step 5 —exitOnError: falseexists solely to prevent Winston from callingprocess.exit(1)beforepsTreeHandlercan suppress the Playwright error. Removing it without also removing the step 5 suppression would cause Winston to exit the process beforepsTreeHandlerruns again.common.ts—getClonedProfilesWithRandomToken: Remove theif (fs.existsSync('/.dockerenv'))block that setsTMPDIR.mergeAxeResults.ts—writeSummaryPdf: Remove theif (process.env.TMPDIR) { fs.emptyDirSync(...) }block.utils.ts—cleanUp: Remove theif (process.env.TMPDIR)removal block.