feat(tar-xz): true streaming for Node extract()/list() — O(largest entry)

[oorabona]

Summary

Closes the TAR-XZ-STREAMING-2026-04-28 story by delivering the "planned optimization" advertised in tar-xz v6.0.0's README. Node extract() and list() now stream — memory stays O(largest single entry) instead of O(archive). For a 1 GB .tar.xz containing 100 MB files, peak heap drops from ~1 GB to ~100 MB.

for await (const entry of extract(bigArchive)) {
  for await (const chunk of entry.data) {
    consumer(chunk);  // chunks arrive as XZ decompresses, no pre-buffer
  }
}

Implementation (5 vertical-slice blocks)

Block	What	Files	Tests
1	streamXz() foundation	xz-helpers.ts	9
2	parseTar() AsyncGenerator coroutine	tar-parser.ts	8
3	extract() rewrite + memoizing helpers	extract.ts	+5
4	list() rewrite	list.ts	+1
5	Security regression + memory-shape gates	test/security.spec.ts (new), test/memory-shape.spec.ts (new), vitest.config.ts, file.ts (TSDoc), README.md	22 + 4

Total: 661 tests passing across all packages (was ~620 pre-PR). tsc clean, lint clean (via rtk proxy biome workaround), build clean.

Spec hardening

The docs/plans/TAR-XZ-STREAMING-2026-04-28.md spec went through:

• /adversarial (5 perspectives) — 13 findings (3S + 6M + 4L); 3S resolved via user-validated decisions A-01 through A-03; 6M+4L folded as spec amendments.
• /llm --spec consensus (Codex + Copilot; Gemini errored on quota) — 9 additional findings; 2 critical S folded as memory-measurement strategy refinement (in-loop high-water sampling) and err.code = 'TAR_PARSER_INVARIANT' Node-convention idiom (the spec's "reuse TarFormatError" was unimplementable — that class did not exist).
• 5 design decisions locked: D-1 chunk pass-through, D-2 hybrid error semantics, D-3 cache helpers + single-use data, D-4 implementation invariants, D-5 invariant error discrimination.

Security

• All 18 TOCTOU vectors closed by PR feat(tar-xz)!: redesign for v6 — universal stream-first API #108 are covered by test/security.spec.ts (22 tests, vector-labeled).
• New PAX-bomb DoS guard: MAX_PAX_HEADER_BYTES = 1 MB — crafted archives declaring giant PAX headers throw TAR_PARSER_INVARIANT instead of accumulating O(N²).
• POSIX path: fd-based with O_NOFOLLOW — TOCTOU window unchanged from buffered model.
• Windows path: by-path fallback's symlink-swap window now scales with entry size (streaming makes it observable). Documented in file.ts @security block + new Security model README subsection. Win32 fd-based extraction tracked as a separate MEDIUM follow-up TODO.

Behavior changes (non-breaking but observable — see changeset)

The new entry.data cadence, bytes()/text() memoization, single-use data, and consumer-break silent-stop are all consistent with the v6.0.0 stream-first contract. Anyone with reasonable reliance on the old buffered behavior should open an issue — we'll ship 7.0.0 with a deprecation cycle if needed.

Test plan

• pnpm build (TypeScript + WASM) — exit 0
• pnpm exec tsc --noEmit — exit 0
• rtk proxy biome lint packages/tar-xz/src/ packages/tar-xz/test/ — exit 0 (warnings unrelated to PR)
• pnpm test — 661/661 passing
• pnpm test:memory --filter @tar-xz — 4/4 passing with --expose-gc available
• Spec doc adversarial+llm-consensus stamped; all locked decisions traceable

Follow-ups (filed in TODO.md)

• [Win32] handle-based extraction (CreateFileW + FILE_FLAG_OPEN_REPARSE_POINT) — close TOCTOU on Windows after this ships.

Story metadata

• Story ID: TAR-XZ-STREAMING-2026-04-28
• Branch: feat/tar-xz-streaming
• Commits: 4 (3 feat + 1 changeset; will be squashed at merge)
• Spec: docs/plans/TAR-XZ-STREAMING-2026-04-28.md
• Bump: tar-xz minor → 6.1.0

[Copilot]

Pull request overview

This PR delivers true streaming for the Node.js tar-xz extract() and list() APIs, replacing the previous buffer-then-parse implementation so memory usage scales with the largest single entry rather than the full archive.

Changes:

• Introduces a streaming XZ decompression helper (streamXz) and a streaming TAR parser coroutine (parseTar).
• Rewrites Node extract()/list() to consume parseTar events and yield entries lazily, with new security + memory-shape regression tests.
• Updates Vitest config/scripts and documentation (README/spec/changeset) to reflect the streaming model and its security implications.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
packages/tar-xz/src/node/xz-helpers.ts	Adds streamXz() streaming decompression pipeline.
packages/tar-xz/src/node/tar-parser.ts	Adds parseTar() async-generator TAR parser + PAX DoS guard.
packages/tar-xz/src/node/extract.ts	Rewrites extract() to be truly streaming; adds memoizing helpers.
packages/tar-xz/src/node/list.ts	Rewrites list() to stream metadata without buffering content.
packages/tar-xz/vitest.config.ts	Forces forks pool + --expose-gc for memory-shape tests.
packages/tar-xz/test/*.spec.ts	Adds/extends streaming, security, and memory-shape tests.
packages/tar-xz/src/node/file.ts	Updates security documentation (POSIX vs Windows TOCTOU model).
packages/tar-xz/README.md	Documents new streaming + security model.
docs/plans/TAR-XZ-STREAMING-2026-04-28.md	Adds design/spec document for the refactor.
packages/tar-xz/package.json	Adds test:memory script.
.changeset/tar-xz-streaming.md	Declares minor release + behavior notes.
TODO.md	Tracks story progress + Windows follow-up.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

parseTar() manually pulls from source[Symbol.asyncIterator]() but never calls iter.return() on early termination (e.g. when the consumer breaks out of extract()/list() and parser.return() is invoked). Adding a try/finally around the main loop to invoke await iter.return?.() would ensure upstream generators/streams (like streamXz()) are properly closed and don’t keep running in the background.

[Copilot]

bytes() currently reads from the same single-use dataGen as entry.data. If a caller iterates entry.data (even partially) and then calls bytes()/text(), the helpers will return only the remaining bytes (or an empty array) while still claiming to “collect all chunks”. Consider either (1) throwing a clear error if data iteration has started before bytes()/text() is called, or (2) implementing a tee/memoization strategy that guarantees bytes() always returns the full content.

[Copilot]

text(encoding?) switched from Buffer.toString(BufferEncoding) semantics to TextDecoder. In Node, TextDecoder supports a narrower set of encoding labels than BufferEncoding (and different aliases), so callers passing encodings like base64, hex, etc. will now throw. If the intent is to preserve the previous Node behavior, decode via Buffer.from(bytes).toString(encoding as BufferEncoding) (or explicitly document/restrict supported encodings).

Suggested change

	return new TextDecoder(encoding ?? 'utf-8').decode(bytes);
	return Buffer.from(bytes).toString((encoding ?? 'utf-8') as BufferEncoding);

[Copilot]

Switching the entire tar-xz test suite to pool: 'forks' can significantly increase test runtime and process overhead vs threads, even though only the memory-shape tests need --expose-gc. Consider using a separate Vitest config (or a dedicated vitest --config ... script) for memory-shape tests so regular pnpm test can keep the default pool.

Suggested change

	export default defineConfig({
	test: {
	// Use forks pool so that --expose-gc (required by memory-shape tests) is
	// available. Memory-shape tests feature-detect gc at runtime and skip if
	// missing, so this flag is non-breaking on CI runners that do not pass it.
	// The forks pool also gives each test file a clean V8 heap.
	//
	// Note: Vitest 4 moved pool-level execArgv to top-level test.execArgv.
	// The previous poolOptions.forks.execArgv is now just test.execArgv.
	pool: 'forks',
	execArgv: ['--expose-gc'],
	include: ['test/**/*.spec.ts'],
	const memoryShapeInclude = ['test/**/*memory-shape*.spec.ts'];
	const runMemoryShapeTests = process.env.VITEST_MEMORY_SHAPE === '1';
	export default defineConfig({
	test: {
	// Only memory-shape tests require --expose-gc. Keep the regular suite on
	// Vitest's default pool, and opt into forks/execArgv only for a dedicated
	// memory-shape run (for example: VITEST_MEMORY_SHAPE=1 vitest).
	...(runMemoryShapeTests
	? {
	pool: 'forks' as const,
	execArgv: ['--expose-gc'],
	include: memoryShapeInclude,
	}
	: {
	include: ['test/**/*.spec.ts'],
	exclude: memoryShapeInclude,
	}),

[Copilot]

Comment has an incorrect 64 KB conversion: 64 KB is 65,536 bytes, not 655,360. Since this comment is used to justify the memory threshold, fixing the number will avoid confusion.

Suggested change

	// stays below 10× the typical XZ output chunk size (64 KB = 655 360 bytes).
	// stays below 10× the typical XZ output chunk size (64 KB = 65,536 bytes).

[Copilot]

streamXz() starts a pipeline() but doesn’t reliably clean it up when the consumer stops iterating early (e.g. extract()/list() consumer breaks). If the async generator is closed before pipelinePromise is awaited, the pipeline can keep running/hang, and a later pipeline rejection can become an unhandled promise rejection. Add try/finally cleanup that aborts/destroys the streams (or uses an AbortController/signal) and always awaits/handles pipelinePromise in finally so early-return is safe.

[Copilot]

README wording here is internally inconsistent: it says Node extract()/list() are streaming as of v6.1.0, but then refers to the previous buffered behavior as the v6.0.0 "planned optimization" delivery. Consider rephrasing to clearly state that v6.1.0 delivers the planned optimization and that v6.0.0 buffered internally.

Suggested change

	- **Stream-shaped API** — all functions return `AsyncIterable<…>`; stream-shaped inputs accepted. Node `extract()`/`list()` are now O(largest entry) streaming as of v6.1.0 — the previous buffered behavior was the v6.0.0 "planned optimization" delivery
	- **Stream-shaped API** — all functions return `AsyncIterable<…>`; stream-shaped inputs accepted. In Node, `extract()`/`list()` are O(largest entry) streaming as of v6.1.0, which delivered the planned optimization; in v6.0.0 they still buffered internally

[Copilot]

This changeset’s description of the pre-v6.1.0 behavior seems inaccurate: the old Node implementation buffered and processed the whole archive before yielding any entries, so "errors after consumer break" wouldn’t have been observable in the same way. Consider tightening the wording to avoid claiming a specific old runtime behavior that didn’t occur with the fully-buffered pipeline.

Suggested change

	These were always part of the streaming contract advertised by v6.0.0 but only become observable now:
	1. **`entry.data` chunk count and size** now reflect the XZ decompressor's output cadence (typically 64 KB blocks). Previously: exactly one chunk per entry containing the full file. Code that assumed "one chunk per entry" or relied on `Buffer.concat` of all chunks should iterate normally — `entry.bytes()` and `entry.text()` still collect the whole file.
	2. **`entry.bytes()` and `entry.text()` memoize.** Calling `bytes()` twice now returns the same buffer reference (efficient) instead of re-collecting. Memory promise preserved: at most one entry's bytes are held at a time.
	3. **`entry.data` is single-use.** Iterating it twice now yields nothing on the second pass (was: same single chunk both times). If you need to re-read, call `bytes()` first and keep the reference.
	4. **Consumer-break with corrupt downstream is now silent.** If you `for await ... break` and the archive is corrupt past the break point, no error surfaces. Previously the buffered model could surface the error after the loop. Errors during forward iteration still throw normally — only post-break errors are suppressed (matches `tar-stream` and JS AsyncGenerator convention).
	These are newly observable consequences of the true-streaming implementation:
	1. **`entry.data` chunk count and size** now reflect the XZ decompressor's output cadence (typically 64 KB blocks). Previously: exactly one chunk per entry containing the full file. Code that assumed "one chunk per entry" or relied on `Buffer.concat` of all chunks should iterate normally — `entry.bytes()` and `entry.text()` still collect the whole file.
	2. **`entry.bytes()` and `entry.text()` memoize.** Calling `bytes()` twice now returns the same buffer reference (efficient) instead of re-collecting. Memory promise preserved: at most one entry's bytes are held at a time.
	3. **`entry.data` is single-use.** Iterating it twice now yields nothing on the second pass (was: same single chunk both times). If you need to re-read, call `bytes()` first and keep the reference.
	4. **Consumer-break with corrupt downstream is now silent.** With true streaming, if you `for await ... break` and the archive is corrupt past the break point, no error surfaces because that later data is never consumed. This differs from the old fully buffered pipeline, which did not expose this same post-break behavior. Errors during forward iteration still throw normally — only corruption beyond the break point is suppressed (matching `tar-stream` and JS AsyncGenerator convention).

[Copilot]

The spec document contains a duplicated section header/content: ## §12 Locked Design Decisions (2026-04-28) appears twice back-to-back. Removing the duplicate will prevent confusion when linking to section anchors and during future edits.

Suggested change

	## §12 Locked Design Decisions (2026-04-28)
	User-validated decisions BEFORE adversarial review (each via separate AskUserQuestion with
	concrete chiffrage):

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

pipelinePromise is created outside the returned async generator, so the pipeline starts as soon as streamXz() is called (even if the caller never iterates). That can eagerly consume the input Readable, do work without backpressure from the consumer, and if the pipeline rejects before iteration begins it can produce an unhandled rejection. Consider creating/starting the pipeline inside the async generator on first iteration (or attaching an immediate .catch() handler at creation time) so the stream stays lazy and errors are always observed by the iterator.

[Copilot]

bytes() currently buffers all chunks into an array and then concatenates them into a new Uint8Array, which doubles copying and peak memory (chunks array + final buffer) for large entries. Since the entry size is known (entry.size), consider allocating the output once and filling it as chunks arrive to keep memory overhead closer to 1× entry size and reduce CPU.

Suggested change

	const chunks: Uint8Array[] = [];
	for await (const c of dataGen) chunks.push(c);
	cachedBytes = concatChunks(chunks);
	const out = new Uint8Array(entry.size);
	let offset = 0;
	for await (const c of dataGen) {
	out.set(c, offset);
	offset += c.byteLength;
	}
	cachedBytes = offset === out.byteLength ? out : out.subarray(0, offset);

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The JSDoc says parseTar “ALWAYS emits a final kind:'end' event before returning”, but the generator can also return early when the consumer stops iteration (generator.return()), in which case no 'end' is yielded. Suggest rewording to “emits 'end' on successful end-of-archive detection” (and mention consumer-abort explicitly) to avoid misleading downstream consumers/tests.

Suggested change

	* The generator ALWAYS emits a final `kind:'end'` event before returning.
	* On successful end-of-archive detection, the generator emits a final
	* `kind:'end'` event before returning. If the consumer stops iteration early
	* (for example via `break` from `for await...of` or `generator.return()`),
	* no final `kind:'end'` event is guaranteed.

[Copilot]

vitest.memory.config.ts sets test.execArgv, but if Vitest doesn’t actually apply this option to the forks pool, globalThis.gc will be missing and the memory-shape suite will silently take the “SKIPPED” path, making pnpm test:memory a no-op. Add a sanity check (fail the run if gc is absent) and/or configure --expose-gc using the supported forks-pool mechanism (e.g. test.poolOptions.forks.execArgv) so the memory gate reliably executes.

Suggested change

	execArgv: ['--expose-gc'],
	poolOptions: {
	forks: {
	execArgv: ['--expose-gc'],
	},
	},