Skip to content

Commit 4d681ab

Browse files
revert(readme): restore canonical README.adoc (#49)
Restores canonical README.adoc and removes the README.md from the over-broad 'convert -> Markdown' pass. AsciiDoc is canonical; .md only on boj-server + hyperpolymath. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 1bc6772 commit 4d681ab

2 files changed

Lines changed: 240 additions & 259 deletions

File tree

README.adoc

Lines changed: 240 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,240 @@
1+
= Docudactyl
2+
image:https://img.shields.io/badge/License-PMPL_1.0-blue.svg[MPL-2.0,link="https://github.com/hyperpolymath/palimpsest-license"]
3+
4+
:toc:
5+
:sectnums:
6+
:source-highlighter: rouge
7+
8+
// Badges
9+
image:https://img.shields.io/badge/RSR-Tier%201-gold[RSR Tier 1]
10+
image:https://img.shields.io/badge/Phase-v0.4.0-green[Phase]
11+
image:https://img.shields.io/badge/Chapel-2.3+-4E9A06?logo=data:image/svg+xml;base64,[Chapel]
12+
image:https://img.shields.io/badge/Zig-0.15+-F7A41D?logo=zig[Zig]
13+
image:https://img.shields.io/badge/Idris2-0.8+-5E5086[Idris2]
14+
image:https://img.shields.io/badge/OCaml-4.14+-EC6813?logo=ocaml[OCaml]
15+
image:https://img.shields.io/badge/Ada-2022-blue[Ada]
16+
17+
== License & Philosophy
18+
19+
This project is licensed under **MPL-2.0** (Palimpsest License).
20+
21+
The full licence text is in `license/PMPL-1.0.txt`. The canonical source is the https://github.com/hyperpolymath/palimpsest-license[palimpsest-license] repository.
22+
23+
== Overview
24+
25+
**Docudactyl** is a multi-format HPC document extraction engine designed for British Library scale (~170 million items). It processes PDFs, images, audio, video, EPUB, and geospatial data across hundreds of cluster nodes.
26+
27+
=== Architecture
28+
29+
[source]
30+
----
31+
┌──────────────────────────────────────────────────────────────────┐
32+
│ Chapel HPC Orchestrator │
33+
│ (64-512 locales, dynamic load balancing) │
34+
├──────────────────────────────────────────────────────────────────┤
35+
│ Conduit │ L1/L2 Cache │ Checkpoint │ Progress Reporter │
36+
│ (validate) │ (LMDB+DFly) │ (resume) │ (ETA, rate) │
37+
├──────────────────────────────────────────────────────────────────┤
38+
│ Zig FFI Layer │
39+
│ (51 C-exported functions, zero overhead) │
40+
├────────┬──────────┬──────────┬──────────┬──────────┬────────────┤
41+
│Poppler │Tesseract │ FFmpeg │ libxml2 │ GDAL │ libvips │
42+
│ (PDF) │ (OCR) │(AV meta) │ (EPUB) │ (Geo) │ (Image) │
43+
├────────┴──────────┴──────────┴──────────┴──────────┴────────────┤
44+
│ dlopen: ONNX Runtime (ML) │ PaddleOCR (GPU OCR) │ CUDA │
45+
├──────────────────────────────────────────────────────────────────┤
46+
│ Idris2 ABI Proofs (14 types, 5 struct layouts, 51 FFI decls) │
47+
└──────────────────────────────────────────────────────────────────┘
48+
49+
Offline: OCaml docudactyl-scm (JSON/text → Scheme S-expressions)
50+
Viewer: Ada TUI (interactive document inspection)
51+
Legacy: Julia extraction scripts (replaced by Chapel pipeline)
52+
----
53+
54+
=== Performance Estimates (British Library, 170M items)
55+
56+
[cols="1,2"]
57+
|===
58+
|Scenario |Estimate
59+
60+
|Cold run (256 nodes + GPU)
61+
|~3.7 hours
62+
63+
|Warm run (L1+L2 cache)
64+
|~4.4 minutes
65+
66+
|Incremental (5% new files)
67+
|~8 minutes
68+
|===
69+
70+
== Quick Start
71+
72+
[source,bash]
73+
----
74+
# Verify dependencies
75+
just deps-check
76+
77+
# Build Zig FFI + Chapel binary
78+
just build-hpc
79+
80+
# Run all tests
81+
just test-hpc
82+
83+
# Process a directory of documents
84+
just generate-manifest /path/to/documents manifest.txt
85+
bin/docudactyl-hpc --manifestPath=manifest.txt --outputDir=output/
86+
87+
# Or on an HPC cluster (64 nodes)
88+
sbatch deploy/slurm-docudactyl.sh
89+
----
90+
91+
== Components
92+
93+
=== Chapel: HPC Engine (hot path)
94+
95+
The Chapel component distributes document processing across cluster nodes with dynamic load balancing.
96+
97+
Modules: Config, ContentType, FFIBridge, ManifestLoader, NdjsonManifest, FaultHandler, ProgressReporter, ShardedOutput, ResultAggregator, Checkpoint, DocudactylHPC.
98+
99+
=== Zig FFI: Parser Dispatch Layer
100+
101+
10 submodules providing a unified C ABI for 7 content types and 20 processing stages:
102+
103+
* **Core**: `docudactyl_ffi.zig` -- init, free, parse, version (dispatches by content type)
104+
* **Stages**: 20 analysis stages with Cap'n Proto output (language, readability, keywords, citations, OCR confidence, perceptual hash, TOC, NER, Whisper, image classify, layout, handwriting, etc.)
105+
* **Cache**: L1 LMDB per-locale (zero-copy mmap) + L2 Dragonfly cross-locale
106+
* **Conduit**: Magic-byte content detection (15 formats), SHA-256, validation
107+
* **GPU OCR**: PaddleOCR CUDA > Tesseract CUDA > CPU (via dlopen)
108+
* **ML Inference**: ONNX Runtime -- NER, Whisper, ImageClassify, Layout, Handwriting (TensorRT > CUDA > OpenVINO > CPU)
109+
* **Hardware Crypto**: SHA-NI, AVX2, AVX-512, AES-NI, ARM SHA2 acceleration
110+
* **I/O Prefetch**: io_uring (Linux 5.6+) with posix_fadvise fallback
111+
112+
=== Idris2: Formal ABI Proofs
113+
114+
Dependent types proving struct layout, alignment, and enum correctness:
115+
116+
* 14 proven types (ContentKind, ParseStatus, MlStatus, MlStage, ExecProvider, Sha256Tier, etc.)
117+
* 5 struct layout proofs (ParseResult 952B, MlResult 48B, CryptoCaps 16B, OcrResult 48B, ConduitResult 88B)
118+
* 51 FFI declarations matching the C header 1:1
119+
120+
=== OCaml: Offline Scheme Transformer
121+
122+
Transforms extracted JSON/text into machine-readable Scheme S-expressions. Not in the HPC hot path.
123+
124+
[source,bash]
125+
----
126+
docudactyl-scm document.pdf -o document.scm
127+
docudactyl-scm extracted.json -o extracted.scm
128+
----
129+
130+
=== Ada: Terminal UI
131+
132+
Interactive viewer for inspecting extracted documents.
133+
134+
[source,bash]
135+
----
136+
docudactyl-tui extracted.json
137+
----
138+
139+
== Justfile Recipes
140+
141+
[source,bash]
142+
----
143+
# Build
144+
just build-hpc # Zig FFI + Chapel binary
145+
just build-ffi # Zig FFI only
146+
just build-idris # Idris2 ABI proofs
147+
just build-ocaml # OCaml transformer
148+
just build-ada # Ada TUI
149+
150+
# Test
151+
just test-hpc # All HPC tests (FFI + error paths)
152+
just test-ffi # Zig integration tests (40+ tests)
153+
just test-scale # Scale test (2105+ files)
154+
just test-idris # Idris2 proofs compile
155+
just test-ocaml # OCaml tests
156+
just test-ada # Ada build check
157+
158+
# Deploy
159+
just deps-check # Verify dependencies
160+
just generate-manifest <dir> [output]
161+
just generate-abi-header
162+
just loc # Lines of code
163+
----
164+
165+
== Directory Structure
166+
167+
[source]
168+
----
169+
docudactyl/
170+
├── src/
171+
│ ├── chapel/ # HPC engine (11 modules)
172+
│ ├── Docudactyl/ABI/ # Idris2 ABI proofs (3 modules)
173+
│ ├── ocaml/ # Offline Scheme transformer
174+
│ ├── ada/ # Terminal UI
175+
│ └── julia/ # Legacy extraction (replaced)
176+
177+
├── ffi/zig/ # Zig FFI layer (10 submodules)
178+
│ ├── src/ # Source
179+
│ └── test/ # Integration tests
180+
181+
├── generated/abi/ # Auto-generated C header
182+
├── schema/ # Cap'n Proto schema
183+
├── deploy/ # Containerfile + Slurm script
184+
├── contractiles/ # K9 contractile configs
185+
├── .machine_readable/ # SCM checkpoint files
186+
├── Justfile # Task runner
187+
└── docudactyl.ipkg # Idris2 package
188+
----
189+
190+
== Requirements
191+
192+
=== System Dependencies
193+
194+
* **Chapel** 2.3+ (HPC engine)
195+
* **Zig** 0.15+ (FFI layer)
196+
* **Idris2** 0.8+ (ABI proofs)
197+
* **C libraries**: Poppler, Tesseract, FFmpeg, libxml2, GDAL, libvips, LMDB
198+
* **Optional**: ONNX Runtime, PaddleOCR, CUDA (for ML/GPU features)
199+
* **OCaml** 4.14+ (offline Scheme transformer)
200+
* **Ada** GNAT/gprbuild (terminal UI)
201+
202+
=== Container Deployment
203+
204+
[source,bash]
205+
----
206+
podman build -f deploy/Containerfile -t docudactyl-hpc .
207+
podman run --rm -v /data/manifest.txt:/manifest.txt:ro \
208+
-v /data/output:/output \
209+
docudactyl-hpc --manifestPath=/manifest.txt
210+
----
211+
212+
=== Cluster Deployment (Slurm)
213+
214+
[source,bash]
215+
----
216+
# Edit deploy/slurm-docudactyl.sh for your cluster
217+
sbatch deploy/slurm-docudactyl.sh
218+
----
219+
220+
== Ethical Use
221+
222+
This tool is designed for:
223+
224+
* Document analysis and archival processing at national library scale
225+
* Research and verification of redaction practices
226+
* Accessibility improvements for PDF content
227+
* Multi-format metadata extraction and cataloguing
228+
229+
== RSR Compliance
230+
231+
This is a Tier 1 RSR project. The hot path uses Chapel + Zig (systems languages). Legacy components (Julia, OCaml, Ada) serve offline/auxiliary roles.
232+
233+
== License
234+
235+
SPDX-License-Identifier: CC-BY-SA-4.0
236+
237+
== Links
238+
239+
* https://github.com/hyperpolymath/docudactyl[GitHub Repository]
240+
* https://rhodium.sh[Rhodium Standard]

0 commit comments

Comments
 (0)