Skip to content

Commit 88f7071

Browse files
hyperpolymathhyperpolymath
committed
docs: substantive CRG C annotation (EXPLAINME.adoc)
1 parent bc24d90 commit 88f7071

1 file changed

Lines changed: 217 additions & 22 deletions

File tree

‎EXPLAINME.adoc‎

Lines changed: 217 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,42 +1,237 @@
11
// SPDX-License-Identifier: PMPL-1.0-or-later
2+
// Copyright (c) 2026 Jonathan D.A. Jewell <j.d.a.jewell@open.ac.uk>
23
= InvestigativeJournalist.jl — Show Me The Receipts
34
:toc:
5+
:toclevels: 3
46
:icons: font
57

6-
The README makes claims. This file backs them up.
8+
The README makes claims. This file backs them up with specific code locations,
9+
honest caveats, and enough detail for an external reviewer to orient themselves.
710

8-
[quote, README]
9-
____
10-
InvestigativeJournalist.jl is a Julia toolkit for managing the end-to-end investigative reporting workflow. It provides high-integrity structures for ingesting diverse sources, extracting verifiable claims, and building a corroboration matrix that minimizes editorial and legal risk.
11-
____
11+
== README Claims — Verified
1212

13-
== Technology Choices
13+
=== Claim 1: "Covers the full investigative pipeline from document ingestion through publication"
1414

15-
[cols="1,2"]
16-
|===
17-
| Technology | Learn More
15+
*True.* `src/InvestigativeJournalist.jl` includes 18 sub-modules spanning the
16+
complete workflow:
1817

19-
| **Zig** | https://ziglang.org
20-
| **Julia** | https://julialang.org
21-
| **Idris2 ABI** | https://www.idris-lang.org
22-
|===
18+
. *Ingestion:* `ingest.jl` (`ingest_source`), `parsers.jl` (`parse_contract`,
19+
`parse_job_description`, `compare_policies`)
20+
. *Analysis:* `claims.jl` (`extract_claim`), `corroboration.jl`
21+
(`link_evidence`, `corroboration_report`), `analytics/statistics.jl`
22+
(`benfords_law_check`, `find_outliers`)
23+
. *Intelligence:* `intelligence/network.jl` (`InvestigativeGraph`,
24+
`add_connection!`, `find_shortest_path`), `intelligence/string_board.jl`
25+
(`CrazyWall`, `add_photo!`, `add_string!`), `intelligence/forensics.jl`
26+
(`verify_image_integrity`, `detect_ai_artifacts`), `intelligence/media_pro.jl`
27+
(`isolate_signal`, `denoise_audio`, `enhance_clarity`), `intelligence/systemic.jl`
28+
(`model_instability_context`, `test_causal_pathway`, `assess_black_swan`)
29+
. *Security:* `shield/unlock.jl` (`unlock_pdf`, `force_extract_text`),
30+
`shield/secure_transfer.jl` (`generate_drop_token`, `sign_evidence_pack`)
31+
. *Publication:* `storytelling/templates.jl` (`build_story_structure`,
32+
`Longform`, `NewsBulletin`, `Thread`), `storytelling/timelines.jl`
33+
(`TimelineBranch`, `TimelineEvent`, `add_event!`, `create_branch`)
34+
. *Interoperability:* `interop/bridge.jl` (`run_r_script`, `export_to_stata`,
35+
`export_to_spss`), `storage/verisim_bridge.jl` (`register_investigation_hexad`,
36+
`vql_query`)
37+
. *Audio:* `audio/production.jl` (`PodcastScript`, `add_segment!`,
38+
`generate_show_notes`)
39+
40+
*Caveat:* The package is at v0.1.0. Several sub-modules (`shield/unlock.jl`,
41+
`audio/production.jl`) rely on external system tools (pdftotext, ffmpeg) being
42+
present in PATH; they are not pure Julia implementations.
43+
44+
=== Claim 2: "Network intelligence graphs and 'crazy wall' string boards"
45+
46+
*True.* `src/intelligence/network.jl` exports `InvestigativeGraph` (backed by
47+
`MetaGraphsNext.jl`), `add_connection!`, and `find_shortest_path`. The graph
48+
supports named entities and typed relationship edges. `src/intelligence/string_board.jl`
49+
exports `CrazyWall` (a mutable struct with photo, string, and note collections)
50+
with `add_photo!` and `add_string!` operations — a programmatic analogue of
51+
the physical investigation corkboard.
52+
53+
Test coverage verifies: entity creation, connection addition, shortest-path
54+
traversal, and CrazyWall item accumulation.
55+
56+
=== Claim 3: "Secure evidence transfer with cryptographic signing"
57+
58+
*True.* `src/shield/secure_transfer.jl` exports `generate_drop_token` and
59+
`sign_evidence_pack`. Both use Julia's `SHA.jl` standard library for
60+
cryptographic hashing and generate self-describing tokens suitable for
61+
secure evidence hand-off. `sign_evidence_pack` produces a detached signature
62+
over a collected evidence bundle.
63+
64+
*Caveat:* The signing scheme uses SHA-256 hashes, not asymmetric signatures
65+
(e.g., Ed25519). This provides integrity verification but not non-repudiation.
66+
Users requiring legal-grade signing should use GnuPG tooling externally.
67+
68+
== How It Works
69+
70+
`src/InvestigativeJournalist.jl` is a module aggregator: it includes 18
71+
sub-module files and re-exports from all 18 using-declarations. The sub-module
72+
architecture isolates concerns — callers needing only network intelligence can
73+
`using InvestigativeJournalist.NetworkIntelligence` without loading the full
74+
package.
75+
76+
The primary data types are `SourceDoc`, `Claim`, `EvidenceLink`, `Entity`,
77+
`Event`, `FOIARequest`, and `StoryDraft` — all defined in `src/types.jl` and
78+
exported from the root module. A corroboration workflow proceeds:
79+
80+
. Ingest source documents via `ingest_source` (returns `SourceDoc` with
81+
SHA-256 fingerprint)
82+
. Extract claims via `extract_claim` (returns `Vector{Claim}`)
83+
. Link supporting evidence via `link_evidence` (attaches `EvidenceLink` to
84+
claims)
85+
. Generate a corroboration matrix via `corroboration_report` (returns
86+
DataFrame of claims with evidence counts and corroboration status)
87+
88+
Network intelligence and forensic analysis run as parallel tracks, feeding
89+
the final `StoryDraft` via `build_story_structure`. VeriSimDB integration
90+
via `storage/verisim_bridge.jl` persists investigations to a structured
91+
queryable store.
92+
93+
== Honest Caveats
94+
95+
* The `detect_ai_artifacts` function in `intelligence/forensics.jl` performs
96+
heuristic analysis; it is not a trained classifier and will produce
97+
false positives on heavily compressed images.
98+
* `run_r_script` and `export_to_stata` in `interop/bridge.jl` spawn subprocesses
99+
and require R and Stata to be installed; they are thin wrappers, not Julia
100+
re-implementations.
101+
* The package has no built-in OCR; `ingest_source` on image PDFs calls an
102+
external pdftotext subprocess via `shield/unlock.jl`.
103+
* VeriSimDB integration (`storage/verisim_bridge.jl`) requires a running
104+
VeriSimDB instance; the bridge will fail gracefully if none is available.
23105

24106
== Dogfooded Across The Account
25107

26-
Uses the hyperpolymath ABI/FFI standard (Idris2 + Zig). Same pattern used across
27-
https://github.com/hyperpolymath/proven[proven],
28-
https://github.com/hyperpolymath/burble[burble], and
29-
https://github.com/hyperpolymath/gossamer[gossamer].
108+
[cols="2,3"]
109+
|===
110+
| Where | How InvestigativeJournalist is relevant
111+
112+
| https://github.com/hyperpolymath/developer-ecosystem[developer-ecosystem]
113+
`julia-ecosystem/`
114+
| InvestigativeJournalist is the most complex multi-module Julia package in
115+
the account and serves as a reference for the sub-module include/using
116+
aggregator pattern.
117+
118+
| https://github.com/hyperpolymath/Causals.jl[Causals.jl]
119+
| IJ's `test_causal_pathway` in `intelligence/systemic.jl` delegates to
120+
Causals.jl `assess_causality` (Bradford Hill) for evaluating causal claims
121+
in investigative stories.
122+
123+
| https://github.com/hyperpolymath/BowtieRisk.jl[BowtieRisk.jl]
124+
| IJ's `assess_black_swan` output can populate threat probability estimates
125+
in a BowtieRisk model for editorial risk assessment.
126+
127+
| https://github.com/hyperpolymath/verisimdb[verisimdb]
128+
| `storage/verisim_bridge.jl` implements `register_investigation_hexad` and
129+
`vql_query` — a direct per-project VeriSimDB instance integration following
130+
the mandatory VeriSimDB policy.
131+
132+
| https://github.com/hyperpolymath/hypatia[hypatia]
133+
| Hypatia CI scans InvestigativeJournalist.jl for SPDX compliance and RSR
134+
antipatterns on every push.
135+
|===
30136

31137
== File Map
32138

33-
[cols="1,2"]
139+
[cols="2,3"]
34140
|===
35-
| Path | What's There
141+
| Path | What is There
142+
143+
| `src/InvestigativeJournalist.jl`
144+
| Root module aggregator. Includes 18 sub-modules and re-exports 40+ public
145+
names covering the full investigative pipeline.
146+
147+
| `src/types.jl`
148+
| Core data types: `SourceDoc`, `Claim`, `EvidenceLink`, `Entity`, `Event`,
149+
`FOIARequest`, `StoryDraft`. All other modules depend on these types.
150+
151+
| `src/ingest.jl`
152+
| Document ingestion: `ingest_source` (returns `SourceDoc` with SHA-256
153+
fingerprint, source type, and timestamp).
154+
155+
| `src/claims.jl`
156+
| Claim extraction: `extract_claim` (returns `Vector{Claim}` from `SourceDoc`).
157+
158+
| `src/corroboration.jl`
159+
| Evidence corroboration: `link_evidence`, `corroboration_report`.
160+
161+
| `src/parsers.jl`
162+
| Structured document parsers: `parse_contract`, `parse_job_description`,
163+
`compare_policies`.
164+
165+
| `src/shield/unlock.jl`
166+
| PDF unlocking and text extraction: `unlock_pdf`, `force_extract_text`.
167+
Shells out to pdftotext.
168+
169+
| `src/shield/secure_transfer.jl`
170+
| Cryptographic evidence handling: `generate_drop_token`, `sign_evidence_pack`
171+
(SHA-256 based).
172+
173+
| `src/intelligence/network.jl`
174+
| Network graph intelligence: `InvestigativeGraph` (MetaGraphsNext backed),
175+
`add_connection!`, `find_shortest_path`.
176+
177+
| `src/intelligence/forensics.jl`
178+
| Media forensics: `verify_image_integrity`, `detect_ai_artifacts`.
179+
180+
| `src/intelligence/string_board.jl`
181+
| CrazyWall corkboard: `CrazyWall`, `add_photo!`, `add_string!`.
182+
183+
| `src/intelligence/systemic.jl`
184+
| Systemic analysis: `model_instability_context`, `test_causal_pathway`
185+
(delegates to Causals.jl), `assess_black_swan`.
186+
187+
| `src/intelligence/media_pro.jl`
188+
| Audio/media signal processing: `isolate_signal`, `denoise_audio`,
189+
`enhance_clarity`.
190+
191+
| `src/audio/production.jl`
192+
| Podcast production: `PodcastScript`, `add_segment!`, `generate_show_notes`.
193+
194+
| `src/interop/bridge.jl`
195+
| Statistical software interoperability: `run_r_script`, `export_to_stata`,
196+
`export_to_spss`.
197+
198+
| `src/storytelling/templates.jl`
199+
| Story structure: `StoryTemplate`, `Longform`, `NewsBulletin`, `Thread`,
200+
`build_story_structure`.
201+
202+
| `src/storytelling/timelines.jl`
203+
| Branching timelines: `TimelineBranch`, `TimelineEvent`, `add_event!`,
204+
`create_branch`.
205+
206+
| `src/analytics/statistics.jl`
207+
| Data anomaly detection: `benfords_law_check`, `find_outliers`.
208+
209+
| `src/storage/verisim_bridge.jl`
210+
| VeriSimDB integration: `register_investigation_hexad`, `vql_query`.
211+
212+
| `test/runtests.jl`
213+
| Tests covering all major types and operations: SourceDoc construction,
214+
Claim extraction, EvidenceLink chaining, corroboration report generation,
215+
network graph traversal, CrazyWall operations, secure token generation,
216+
story template construction, and timeline branching.
217+
218+
| `benches/benchmarks.jl`
219+
| BenchmarkTools suite for `ingest_source`, `corroboration_report`, and
220+
network graph shortest-path at varying graph sizes.
221+
222+
| `Project.toml`
223+
| Name `InvestigativeJournalist`, v0.1.0, MPL-2.0; deps include `DataFrames`,
224+
`JSON3`, `SQLite`, `Dates`, `SHA`, `HTTP`, `URIs`, `TextAnalysis`, `Images`,
225+
`Graphs`, `MetaGraphsNext`, `Gumbo`, `Cascadia`.
226+
227+
| `generated/`
228+
| Auto-generated content directory (story exports, evidence bundles).
229+
230+
| `contractiles/`
231+
| Contractile interface definitions (RSR interoperability standard).
36232

37-
| `src/` | Source code
38-
| `ffi/` | Foreign function interface
39-
| `test(s)/` | Test suite
233+
| `stapeln.toml`
234+
| Stapeln container configuration for reproducible build environment.
40235
|===
41236

42237
== Questions?

0 commit comments

Comments
 (0)