Skip to content
This repository was archived by the owner on Jul 11, 2026. It is now read-only.

Commit 98a0b0e

Browse files
h4x0rclaude
andcommitted
docs(task): intra-image enabler + inter-image ingest DONE (strict TDD)
Records the implemented pieces (reader Sync positioned reads b1d8be8/37c2963; issen parse_sources_parallel 94b3f7d/d9c0ac5 + ingest wiring a38cf95) and the remaining TODO: a parallel consumer in ewf (verify/hash) and the issen-ewf wrapper drop-Mutex (gated on publishing ewf 0.2.2). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 37c2963 commit 98a0b0e

1 file changed

Lines changed: 32 additions & 1 deletion

File tree

docs/parallel-chunk-decompression-task.md

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,37 @@
11
# Task: Parallel chunk-decompression (intra-image) + multi-source ingest (inter-image)
22

3-
Status: **planned** (fleet task). Research done 2026-06-27. Build with **strict TDD** + before/after benchmark on a real E01.
3+
Status: **partially DONE** (2026-06-27). The reader concurrency *enabler* and the
4+
issen inter-image ingest parallelism are implemented with strict TDD; the remaining
5+
piece is a parallel *consumer* in ewf (parallel verify/hash) to cash in the intra-image
6+
win for full-image ops. Research done 2026-06-27.
7+
8+
## Done so far (strict TDD, separate RED/GREEN commits)
9+
10+
- **Intra-image enabler — `EwfReader` is now `Sync` with positioned reads** (ewf repo):
11+
`read_at(&self)` serves cursor-free positioned reads (`pread`/`seek_read`, no mmap, keeps
12+
`forbid(unsafe)`); chunk cache is `Mutex<LruCache>` updated through `&self`, decompression
13+
happens *outside* the lock so distinct chunks decompress in parallel. RED `b1d8be8`
14+
(8-thread disjoint + 16-thread hot-chunk differential vs serial) → GREEN `37c2963`.
15+
Validated: full ewf suite + the `ewfexport` differential oracle stays byte-correct.
16+
- **Inter-image — parallel multi-source ingest** (issen repo): `parse_sources_parallel`
17+
(capped, order-preserving rayon map) RED `94b3f7d` (order + `Barrier(2)` concurrency proof
18+
+ clamp) → GREEN `d9c0ac5`; wired into `ingest::run` as phase A (serial store setup +
19+
resume) / phase B (parallel parse, cap = cores-2) / phase C (serial in-order commit) —
20+
commit `a38cf95`. Determinism holds because cross-source events differ in
21+
`evidence_source_id``record_hash`, so the `ORDER BY timestamp_ns, record_hash, id` sort
22+
is run-order-independent. Validated: real DC01 E01 still ingests 1,072,067 events; issen-cli
23+
lib 150/150; integration suite byte-identical to HEAD (the 36 fails are pre-existing).
24+
25+
## Still TODO
26+
27+
1. **Parallel consumer in ewf** — rayon over the chunk range inside `verify()` / a `par_read`,
28+
so `ewf verify` and full-image hashers actually use the now-`Sync` reader (the intra-image
29+
benchmark below). This is the remaining intra-image payoff.
30+
2. **issen-ewf wrapper** — drop its `Mutex<EwfReader>`, delegate to `read_at(&self)`; gated on
31+
**publishing ewf 0.2.2** (issen depends on the registry `ewf = "0.2"`, where `read_at` is
32+
still private). Bump + publish ewf 0.2.2, then `cargo update -p ewf` in issen.
33+
34+
Build the remainder with **strict TDD** + before/after benchmark on a real E01.
435

536
## Why (and why NOT for issen's ingest)
637

0 commit comments

Comments
 (0)