Skip to content

Commit 91d4a48

Browse files
hyperpolymathhyperpolymath
committed
Add sponsor badge
1 parent 0900ca2 commit 91d4a48

1 file changed

Lines changed: 242 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 242 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,242 @@
1+
[![Sponsor](https://img.shields.io/badge/Sponsor-%E2%9D%A4-pink?logo=github)](https://github.com/sponsors/hyperpolymath)
2+
3+
= Docudactyl
4+
image:https://img.shields.io/badge/License-PMPL_1.0-blue.svg[MPL-2.0,link="https://github.com/hyperpolymath/palimpsest-license"]
5+
6+
:toc:
7+
:sectnums:
8+
:source-highlighter: rouge
9+
10+
// Badges
11+
image:https://img.shields.io/badge/RSR-Tier%201-gold[RSR Tier 1]
12+
image:https://img.shields.io/badge/Phase-v0.4.0-green[Phase]
13+
image:https://img.shields.io/badge/Chapel-2.3+-4E9A06?logo=data:image/svg+xml;base64,[Chapel]
14+
image:https://img.shields.io/badge/Zig-0.15+-F7A41D?logo=zig[Zig]
15+
image:https://img.shields.io/badge/Idris2-0.8+-5E5086[Idris2]
16+
image:https://img.shields.io/badge/OCaml-4.14+-EC6813?logo=ocaml[OCaml]
17+
image:https://img.shields.io/badge/Ada-2022-blue[Ada]
18+
19+
== License & Philosophy
20+
21+
This project is licensed under **MPL-2.0** (Palimpsest License).
22+
23+
The full licence text is in `license/PMPL-1.0.txt`. The canonical source is the https://github.com/hyperpolymath/palimpsest-license[palimpsest-license] repository.
24+
25+
== Overview
26+
27+
**Docudactyl** is a multi-format HPC document extraction engine designed for British Library scale (~170 million items). It processes PDFs, images, audio, video, EPUB, and geospatial data across hundreds of cluster nodes.
28+
29+
=== Architecture
30+
31+
[source]
32+
----
33+
┌──────────────────────────────────────────────────────────────────┐
34+
│ Chapel HPC Orchestrator │
35+
│ (64-512 locales, dynamic load balancing) │
36+
├──────────────────────────────────────────────────────────────────┤
37+
│ Conduit │ L1/L2 Cache │ Checkpoint │ Progress Reporter │
38+
│ (validate) │ (LMDB+DFly) │ (resume) │ (ETA, rate) │
39+
├──────────────────────────────────────────────────────────────────┤
40+
│ Zig FFI Layer │
41+
│ (51 C-exported functions, zero overhead) │
42+
├────────┬──────────┬──────────┬──────────┬──────────┬────────────┤
43+
│Poppler │Tesseract │ FFmpeg │ libxml2 │ GDAL │ libvips │
44+
│ (PDF) │ (OCR) │(AV meta) │ (EPUB) │ (Geo) │ (Image) │
45+
├────────┴──────────┴──────────┴──────────┴──────────┴────────────┤
46+
│ dlopen: ONNX Runtime (ML) │ PaddleOCR (GPU OCR) │ CUDA │
47+
├──────────────────────────────────────────────────────────────────┤
48+
│ Idris2 ABI Proofs (14 types, 5 struct layouts, 51 FFI decls) │
49+
└──────────────────────────────────────────────────────────────────┘
50+
51+
Offline: OCaml docudactyl-scm (JSON/text → Scheme S-expressions)
52+
Viewer: Ada TUI (interactive document inspection)
53+
Legacy: Julia extraction scripts (replaced by Chapel pipeline)
54+
----
55+
56+
=== Performance Estimates (British Library, 170M items)
57+
58+
[cols="1,2"]
59+
|===
60+
|Scenario |Estimate
61+
62+
|Cold run (256 nodes + GPU)
63+
|~3.7 hours
64+
65+
|Warm run (L1+L2 cache)
66+
|~4.4 minutes
67+
68+
|Incremental (5% new files)
69+
|~8 minutes
70+
|===
71+
72+
== Quick Start
73+
74+
[source,bash]
75+
----
76+
# Verify dependencies
77+
just deps-check
78+
79+
# Build Zig FFI + Chapel binary
80+
just build-hpc
81+
82+
# Run all tests
83+
just test-hpc
84+
85+
# Process a directory of documents
86+
just generate-manifest /path/to/documents manifest.txt
87+
bin/docudactyl-hpc --manifestPath=manifest.txt --outputDir=output/
88+
89+
# Or on an HPC cluster (64 nodes)
90+
sbatch deploy/slurm-docudactyl.sh
91+
----
92+
93+
== Components
94+
95+
=== Chapel: HPC Engine (hot path)
96+
97+
The Chapel component distributes document processing across cluster nodes with dynamic load balancing.
98+
99+
Modules: Config, ContentType, FFIBridge, ManifestLoader, NdjsonManifest, FaultHandler, ProgressReporter, ShardedOutput, ResultAggregator, Checkpoint, DocudactylHPC.
100+
101+
=== Zig FFI: Parser Dispatch Layer
102+
103+
10 submodules providing a unified C ABI for 7 content types and 20 processing stages:
104+
105+
* **Core**: `docudactyl_ffi.zig` -- init, free, parse, version (dispatches by content type)
106+
* **Stages**: 20 analysis stages with Cap'n Proto output (language, readability, keywords, citations, OCR confidence, perceptual hash, TOC, NER, Whisper, image classify, layout, handwriting, etc.)
107+
* **Cache**: L1 LMDB per-locale (zero-copy mmap) + L2 Dragonfly cross-locale
108+
* **Conduit**: Magic-byte content detection (15 formats), SHA-256, validation
109+
* **GPU OCR**: PaddleOCR CUDA > Tesseract CUDA > CPU (via dlopen)
110+
* **ML Inference**: ONNX Runtime -- NER, Whisper, ImageClassify, Layout, Handwriting (TensorRT > CUDA > OpenVINO > CPU)
111+
* **Hardware Crypto**: SHA-NI, AVX2, AVX-512, AES-NI, ARM SHA2 acceleration
112+
* **I/O Prefetch**: io_uring (Linux 5.6+) with posix_fadvise fallback
113+
114+
=== Idris2: Formal ABI Proofs
115+
116+
Dependent types proving struct layout, alignment, and enum correctness:
117+
118+
* 14 proven types (ContentKind, ParseStatus, MlStatus, MlStage, ExecProvider, Sha256Tier, etc.)
119+
* 5 struct layout proofs (ParseResult 952B, MlResult 48B, CryptoCaps 16B, OcrResult 48B, ConduitResult 88B)
120+
* 51 FFI declarations matching the C header 1:1
121+
122+
=== OCaml: Offline Scheme Transformer
123+
124+
Transforms extracted JSON/text into machine-readable Scheme S-expressions. Not in the HPC hot path.
125+
126+
[source,bash]
127+
----
128+
docudactyl-scm document.pdf -o document.scm
129+
docudactyl-scm extracted.json -o extracted.scm
130+
----
131+
132+
=== Ada: Terminal UI
133+
134+
Interactive viewer for inspecting extracted documents.
135+
136+
[source,bash]
137+
----
138+
docudactyl-tui extracted.json
139+
----
140+
141+
== Justfile Recipes
142+
143+
[source,bash]
144+
----
145+
# Build
146+
just build-hpc # Zig FFI + Chapel binary
147+
just build-ffi # Zig FFI only
148+
just build-idris # Idris2 ABI proofs
149+
just build-ocaml # OCaml transformer
150+
just build-ada # Ada TUI
151+
152+
# Test
153+
just test-hpc # All HPC tests (FFI + error paths)
154+
just test-ffi # Zig integration tests (40+ tests)
155+
just test-scale # Scale test (2105+ files)
156+
just test-idris # Idris2 proofs compile
157+
just test-ocaml # OCaml tests
158+
just test-ada # Ada build check
159+
160+
# Deploy
161+
just deps-check # Verify dependencies
162+
just generate-manifest <dir> [output]
163+
just generate-abi-header
164+
just loc # Lines of code
165+
----
166+
167+
== Directory Structure
168+
169+
[source]
170+
----
171+
docudactyl/
172+
├── src/
173+
│ ├── chapel/ # HPC engine (11 modules)
174+
│ ├── Docudactyl/ABI/ # Idris2 ABI proofs (3 modules)
175+
│ ├── ocaml/ # Offline Scheme transformer
176+
│ ├── ada/ # Terminal UI
177+
│ └── julia/ # Legacy extraction (replaced)
178+
│
179+
├── ffi/zig/ # Zig FFI layer (10 submodules)
180+
│ ├── src/ # Source
181+
│ └── test/ # Integration tests
182+
│
183+
├── generated/abi/ # Auto-generated C header
184+
├── schema/ # Cap'n Proto schema
185+
├── deploy/ # Containerfile + Slurm script
186+
├── contractiles/ # K9 contractile configs
187+
├── .machine_readable/ # SCM checkpoint files
188+
├── Justfile # Task runner
189+
└── docudactyl.ipkg # Idris2 package
190+
----
191+
192+
== Requirements
193+
194+
=== System Dependencies
195+
196+
* **Chapel** 2.3+ (HPC engine)
197+
* **Zig** 0.15+ (FFI layer)
198+
* **Idris2** 0.8+ (ABI proofs)
199+
* **C libraries**: Poppler, Tesseract, FFmpeg, libxml2, GDAL, libvips, LMDB
200+
* **Optional**: ONNX Runtime, PaddleOCR, CUDA (for ML/GPU features)
201+
* **OCaml** 4.14+ (offline Scheme transformer)
202+
* **Ada** GNAT/gprbuild (terminal UI)
203+
204+
=== Container Deployment
205+
206+
[source,bash]
207+
----
208+
podman build -f deploy/Containerfile -t docudactyl-hpc .
209+
podman run --rm -v /data/manifest.txt:/manifest.txt:ro \
210+
-v /data/output:/output \
211+
docudactyl-hpc --manifestPath=/manifest.txt
212+
----
213+
214+
=== Cluster Deployment (Slurm)
215+
216+
[source,bash]
217+
----
218+
# Edit deploy/slurm-docudactyl.sh for your cluster
219+
sbatch deploy/slurm-docudactyl.sh
220+
----
221+
222+
== Ethical Use
223+
224+
This tool is designed for:
225+
226+
* Document analysis and archival processing at national library scale
227+
* Research and verification of redaction practices
228+
* Accessibility improvements for PDF content
229+
* Multi-format metadata extraction and cataloguing
230+
231+
== RSR Compliance
232+
233+
This is a Tier 1 RSR project. The hot path uses Chapel + Zig (systems languages). Legacy components (Julia, OCaml, Ada) serve offline/auxiliary roles.
234+
235+
== License
236+
237+
SPDX-License-Identifier: MPL-2.0
238+
239+
== Links
240+
241+
* https://github.com/hyperpolymath/docudactyl[GitHub Repository]
242+
* https://rhodium.sh[Rhodium Standard]

0 commit comments

Comments
 (0)