feat(tar-xz): true streaming for Node extract()/list() — O(largest entry)

.changeset/tar-xz-streaming.md

@@ -0,0 +1,42 @@
+---
+'tar-xz': minor
+---
+
+True streaming for Node `extract()` and `list()` — memory now stays O(largest single entry) instead of O(archive). Delivers the "planned optimization" advertised by v6.0.0's stream-first README.
+
+```ts
+// 1 GB .tar.xz with 100 MB files → peak heap ~100 MB, not ~1 GB
+for await (const entry of extract(bigArchive)) {
+  for await (const chunk of entry.data) {
+    consumer(chunk);  // chunks arrive as XZ decompresses, no pre-buffer
+  }
+}
+```
+
+## What changed under the hood
+
+- New shared `parseTar(source, mode)` AsyncGenerator coroutine in `tar-parser.ts`. The old `TarUnpack`/`TarList` `Writable` classes are gone; they were artificially-buffered incremental parsers that never received true streaming input.
+- `extract()` and `list()` rewired to consume `parseTar` directly via `for await`, with the consumer's iteration controlling decompression rate (natural backpressure — no explicit pause/resume).
+- Deprecated helpers `collectAllChunks` / `decompressXz` / `runWritable` removed from `xz-helpers.ts` (replaced by `streamXz()`).
+- New PAX-bomb DoS guard: `MAX_PAX_HEADER_BYTES = 1 MB`. Crafted archives declaring giant PAX headers throw `Error` with `code === 'TAR_PARSER_INVARIANT'`.
+
+## Behavior changes (non-breaking but observable — Hyrum's Law)
+
+These were always part of the streaming contract advertised by v6.0.0 but only become observable now:
+
+1. **`entry.data` chunk count and size** now reflect the XZ decompressor's output cadence (typically 64 KB blocks). Previously: exactly one chunk per entry containing the full file. Code that assumed "one chunk per entry" or relied on `Buffer.concat` of all chunks should iterate normally — `entry.bytes()` and `entry.text()` still collect the whole file.
+2. **`entry.bytes()` and `entry.text()` memoize.** Calling `bytes()` twice now returns the same buffer reference (efficient) instead of re-collecting. Memory promise preserved: at most one entry's bytes are held at a time.
+3. **`entry.data` is single-use.** Iterating it twice now yields nothing on the second pass (was: same single chunk both times). If you need to re-read, call `bytes()` first and keep the reference.
+4. **Consumer-break stops iteration cleanly.** If you `for await ... break`, the generator closes and no further errors are surfaced — whether or not the archive is corrupt past the break point. This matches the JS AsyncGenerator convention (`tar-stream`, `it-tar`) where errors beyond the consumer's read horizon are silently discarded. Errors during active forward iteration still throw normally.
+
+If any of these changes break your code, please open an issue — we're prepared to ship 7.0.0 with a deprecation cycle if a reasonable reliance is demonstrated.
+
+## Security
+
+- POSIX path remains fd-based with `O_NOFOLLOW` — TOCTOU window unchanged.
+- Windows fallback: by-path operations, so the symlink-swap window now scales with entry size in streaming mode. Mitigation: extract to a directory not writable by other processes. Win32 fd-based extraction tracked as a separate follow-up.
+- See `Security model` section in the README for details.
+
+## Native parity (out of scope this release)
+
+The streaming refactor only affects the Node path. The browser/WASM path uses a different decompression model and remains buffered (a separate optimization).

TODO.md

@@ -2,15 +2,25 @@
 
 ## In Progress
 
-_None_
+- [ ] 🟡 [tar-xz] True streaming for Node `extract()`/`list()` — Priority: M (started 2026-04-28, branch feat/tar-xz-streaming, story TAR-XZ-STREAMING-2026-04-28)
+  - [x] ✅ Block 1: `streamXz()` added to `xz-helpers.ts`, old helpers `@deprecated`-tagged, 9 tests in `test/xz-helpers.spec.ts` — all quality gates pass (2026-04-28)
+  - [x] ✅ Block 2: `parseTar()` AsyncGenerator + `ParseEvent` type added to `tar-parser.ts`; v8-ignore removed from lines 86-89/148-153/169-174 (now hot paths); 8 tests in `test/tar-parser-stream.spec.ts` covering 128-byte chunking, PAX split (S-05), PAX_GLOBAL split (L-M-03), EOA detection, truncation (S-09), list/no-chunk, auto-drain, TAR_PARSER_INVARIANT (D-5); MAX_PAX_HEADER_BYTES=1MB DoS guard (A-07); all gates pass (2026-04-28)
+  - [x] ✅ Block 3: `extract.ts` rewritten — `TarUnpack` class replaced by lean async generator using `parseTar`; `makeTarEntryWithData` accepts pull-callback with bytes() memoization (D-3); lookahead buffer pattern for correct event routing; S-08 auto-drain + S-08b consumer-break + memoization tests added to `coverage.spec.ts`; all gates pass (2026-04-28)
+  - [x] ✅ Block 4: `list.ts` rewritten — `TarList` class replaced by 4-line generator using `parseTar(xzStream, 'list')`; S-12 placeholder test added; all gates pass (2026-04-28)
+  - [x] ✅ Cleanup: `collectAllChunks`, `decompressXz`, `runWritable` removed from `xz-helpers.ts` (zero callers verified); T-06 deprecated-helpers test removed from `xz-helpers.spec.ts`; all gates pass (2026-04-28)
+  - [x] ✅ Block 5: security regression gate (`test/security.spec.ts` — 18 TOCTOU vectors + S-14 + S-15) + memory-shape CI gate (`test/memory-shape.spec.ts` — 3 high-water tests); vitest.config.ts pool=forks+--expose-gc; file.ts @security TSDoc; README security model subsection (2026-04-28)
+  - [x] ✅ PR #113 Fix Round 1: 11 findings fixed (F-1 streamXz/parseTar early-termination cleanup; F-2 bytes()-after-iter throw guard; F-3 text() reverted to Buffer.toString for base64/hex/latin1 support; F-4 concurrent dataGen guard; F-5 stray-chunk TAR_PARSER_INVARIANT throw; F-6 memory-shape Test1 preset:6; F-7 vitest.memory.config.ts + test:memory script isolated; F-8 math comment fix; F-9 README wording; F-10 changeset wording; F-11 duplicate §12 header removed); 12 new tests; all 6 quality gates green (2026-04-28)
+  - [x] ✅ PR #113 Fix Round 2: CR2-1 (M) streamXz() lazy pipeline — moved pipeline()/createUnxz() inside async generator body so no I/O starts before first .next(); T-07 lazy test (readCount=0 + no unhandled rejection); CR2-2 (L) bytes() alloc-once via entry.size — single Uint8Array pre-alloc + in-place set() per chunk, throws TAR_PARSER_INVARIANT on overrun, special-cases entry.size===0; concatChunks() dead-code deleted; all 6 quality gates green: build=0 tsc=0 lint=0 test=0(151) memory=0(3) full=0(489) (2026-04-29)
+  - [x] ✅ PR #113 CI hotfix: dataWrapper retyped from `AsyncGenerator<Uint8Array>` to `AsyncIterable<Uint8Array>` in extract.ts:makeTarEntryWithData — avoids the `[Symbol.asyncDispose]` requirement TS 6 added to AsyncGenerator (lib.esnext Explicit Resource Management). Wrapper restructured: pure AsyncIterable whose `[Symbol.asyncIterator]()` sets dataIterStarted flag and returns dataGen directly (smaller surface, exactly matches public TarEntryWithData.data type). All 6 gates green. (2026-04-29)
 
 ## Pending - HIGH
 
 _None_
 
 ## Pending - MEDIUM
 
-- [ ] [tar-xz] True streaming for Node `extract()`/`list()` — replace `Buffer.concat` accumulation (extract.ts:59,91 + list.ts:26) with incremental header→content parsing so memory stays O(largest entry) instead of O(archive). Public README v6.0.0 advertises this as a "planned optimization" — Priority: M
+- [ ] [tar-xz] True streaming for Node `extract()`/`list()` — replace `Buffer.concat` accumulation (extract.ts:59,91 + list.ts:26) with incremental header→content parsing so memory stays O(largest entry) instead of O(archive). Public README v6.0.0 advertises this as a "planned optimization" — Priority: M (now in progress as TAR-XZ-STREAMING-2026-04-28 / branch feat/tar-xz-streaming)
+- [ ] [Win32] Handle-based extraction (CreateFileW + FILE_FLAG_OPEN_REPARSE_POINT) for `tar-xz` Node `extractFile` — close the TOCTOU window on Windows surfaced by streaming refactor in PR #113. POSIX uses fd-based O_NOFOLLOW; Windows currently falls back to by-path `pipeline(Readable.from(entry.data), createWriteStream(target))` which exposes a wallclock-long symlink-swap window. Estimate ~1-2h. Match node-tar's gradual approach. Priority: M (post-#113).
 
 ## Pending - LOW (Nice to Have)