docs: tighten README structure and sample references (#18)

* docs: clean up README and add sample fixtures

* docs: expand README diagrams and sample indexes

* docs: refine README audiences and CLI diagrams

* fix: link compliance gate artifacts in PR comments

---------

Co-authored-by: b <b@elastic.ventures>

Author: elasticdotventures

.github/workflows/compliance-gate.yml

@@ -86,6 +86,7 @@ jobs:
           category: compliance-gate
 
       - name: Upload evidence store artifacts
+        id: upload_evidence
         if: always()
         uses: actions/upload-artifact@v4
         with:
@@ -99,6 +100,9 @@ jobs:
       - name: Comment on PR with compliance results
         if: github.event_name == 'pull_request' && always()
         uses: actions/github-script@v7
+        env:
+          WORKFLOW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+          EVIDENCE_ARTIFACT_URL: ${{ steps.upload_evidence.outputs.artifact-url }}
         with:
           script: |
             const fs = require('fs');
@@ -133,6 +137,8 @@ jobs:
             const conditional = results.filter(r => r.status === 'conditional_pass').length;
             const total = results.length;
             const statusIcon = summary.gate_status === 'passed' ? '✅' : '❌';
+            const workflowRunUrl = process.env.WORKFLOW_RUN_URL;
+            const evidenceArtifactUrl = process.env.EVIDENCE_ARTIFACT_URL;
 
             const formatIssues = (title, issues) => {
               if (!issues || issues.length === 0) {
@@ -151,6 +157,13 @@ jobs:
                 ).join('\n')
               : '| none | n/a | n/a |';
 
+            const links = [
+              `- [Workflow run](${workflowRunUrl})`,
+              evidenceArtifactUrl
+                ? `- [Compliance evidence artifact](${evidenceArtifactUrl})`
+                : '- Compliance evidence artifact: not available for this run',
+            ].join('\n');
+
             const comment = `## ${statusIcon} Compliance Gate Results
 
             **Summary:**
@@ -179,7 +192,10 @@ jobs:
             |-------------|--------|-------|
             ${detailRows}
 
-            See artifacts for complete SARIF reports, summary JSON, and evidence.
+            **Artifacts and run links:**
+            ${links}
+
+            The compliance evidence artifact contains the complete SARIF reports, summary JSON, and evidence bundle for this run.
             `;
 
             await github.rest.issues.createComment({

.gitignore

@@ -211,3 +211,41 @@ __marimo__/
 
 # Evidence store - runtime artifacts from compliance gate evaluations
 evidence_store/
+
+# Logs
+logs
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+dev-debug.log
+# Dependency directories
+node_modules/
+# Environment variables
+# Editor directories and files
+.idea
+.vscode
+*.suo
+*.ntvs*
+*.njsproj
+*.sln
+*.sw?
+# OS specific
+.DS_Store
+
+# Task files
+# tasks.json
+# tasks/ 
+
+# Local agent/task scratch state
+.env.example
+.ralph/
+.ralphrc
+.taskmaster/
+tmp/
+Downloads/
+README.md.orig
+_b00t_/
+prd-reqif-ingest/
+ralph/.archive/
+tasks/aemo-stack.prompt.md
+*:Zone.Identifier

README-reqif-ingest-cli.md

@@ -3,6 +3,24 @@
 This repo now includes a standalone ingestion surface at `reqif_ingest_cli/`.
 It keeps source-document intake separate from the existing `reqif_mcp` ReqIF parser and policy server.
 
+Executive view:
+
+- this is the deterministic artifact-to-ReqIF derivation pipeline
+- its purpose is traceable extraction, not policy judgement
+
+Engineer view:
+
+- use this surface when the starting point is an artifact such as XLSX, PDF, DOCX, or Markdown
+- use `reqif_mcp` when the starting point is already ReqIF
+
+```mermaid
+flowchart LR
+    ART[Source artifact] --> EXT[Deterministic extraction]
+    EXT --> DG[document_graph]
+    DG --> CAND[requirement_candidate]
+    CAND --> REQIF[Derived ReqIF]
+```
+
 ## Scope
 
 - Deterministic first pass only.
@@ -12,6 +30,15 @@ It keeps source-document intake separate from the existing `reqif_mcp` ReqIF par
 - ReqIF stays derived: `artifact -> document_graph -> requirement_candidate -> reqif`.
 - Optional LLM quality-eval hooks use an Azure Foundry/OpenAI-compatible adapter and are disabled by default.
 
+```mermaid
+flowchart TD
+    INPUT[Artifact input] --> HASH[Register hash and metadata]
+    HASH --> STRUCTURE[Extract structure]
+    STRUCTURE --> DISTILL[Deterministic distillation]
+    DISTILL --> OUTPUT[Emit derived ReqIF]
+    OUTPUT -. optional review .-> LLM[Foundry quality hook]
+```
+
 ## Commands
 
 Use the local justfile:
@@ -20,18 +47,28 @@ Use the local justfile:
 just -f reqif_ingest_cli/justfile --list
 ```
 
+```mermaid
+flowchart LR
+    TEST[test] --> LINT[lint]
+    LINT --> TYPE[typecheck]
+    TYPE --> ARTIFACT[artifact]
+    ARTIFACT --> EXTRACT[extract]
+    EXTRACT --> DISTILL[distill]
+    DISTILL --> EMIT[emit]
+```
+
 Common commands:
 
 ```bash
 just -f reqif_ingest_cli/justfile test
 just -f reqif_ingest_cli/justfile lint
 just -f reqif_ingest_cli/justfile typecheck
 
-just -f reqif_ingest_cli/justfile artifact "The AESCSF v2 Core.xlsx"
-just -f reqif_ingest_cli/justfile extract "The AESCSF v2 Core.xlsx"
-just -f reqif_ingest_cli/justfile distill "The AESCSF v2 Core.xlsx"
+just -f reqif_ingest_cli/justfile artifact "samples/aemo/The AESCSF v2 Core.xlsx"
+just -f reqif_ingest_cli/justfile extract "samples/aemo/The AESCSF v2 Core.xlsx"
+just -f reqif_ingest_cli/justfile distill "samples/aemo/The AESCSF v2 Core.xlsx"
 just -f reqif_ingest_cli/justfile emit \
-  "The AESCSF v2 Core.xlsx" \
+  "samples/aemo/The AESCSF v2 Core.xlsx" \
   "evidence_store/toolkits/aemo/aescsf-core.reqif" \
   auto \
   "AESCSF Core Derived Baseline"
@@ -49,26 +86,66 @@ just -f reqif_ingest_cli/justfile smoke-aemo-toolkit
 The standalone module runs directly with `uv`:
 
 ```bash
-uv run python -m reqif_ingest_cli register-artifact "The AESCSF v2 Core.xlsx" --pretty
-uv run python -m reqif_ingest_cli extract "The AESCSF v2 Core.xlsx" --pretty
-uv run python -m reqif_ingest_cli distill "The AESCSF v2 Core.xlsx" --pretty
+uv run python -m reqif_ingest_cli register-artifact "samples/aemo/The AESCSF v2 Core.xlsx" --pretty
+uv run python -m reqif_ingest_cli extract "samples/aemo/The AESCSF v2 Core.xlsx" --pretty
+uv run python -m reqif_ingest_cli distill "samples/aemo/The AESCSF v2 Core.xlsx" --pretty
 uv run python -m reqif_ingest_cli emit-reqif \
-  "The AESCSF v2 Core.xlsx" \
+  "samples/aemo/The AESCSF v2 Core.xlsx" \
   --title "AESCSF Core Derived Baseline" \
   --output "evidence_store/toolkits/aemo/aescsf-core.reqif" \
   --pretty
 uv run python -m reqif_ingest_cli foundry-config --pretty
 ```
 
+```mermaid
+sequenceDiagram
+    participant User
+    participant CLI as reqif_ingest_cli
+    participant FS as source artifact
+    participant Out as JSON / ReqIF output
+
+    User->>CLI: register-artifact
+    CLI->>FS: hash + inspect metadata
+    User->>CLI: extract
+    CLI->>Out: document_graph
+    User->>CLI: distill
+    CLI->>Out: requirement_candidate
+    User->>CLI: emit-reqif
+    CLI->>Out: derived ReqIF
+```
+
 ## What It Emits
 
 - `artifact/1`: immutable source hash, media type, file format, source path, profile
 - `document_graph/1`: sections, rows, paragraphs, anchors, semantic IDs
 - `requirement_candidate/1`: deterministic candidate text, rationale, rule ID, provenance
 - ReqIF XML: minimal derived baseline that round-trips through the current parser
 
+```mermaid
+flowchart LR
+    A[artifact/1] --> G[document_graph/1]
+    G --> C[requirement_candidate/1]
+    C --> R[ReqIF XML]
+```
+
+See also:
+
+- `samples/README.md`
+- `samples/aemo/README.md`
+- `samples/contracts/README.md`
+
 ## Current Profiles
 
+```mermaid
+flowchart LR
+    XLSX[XLSX] --> CORE[aescsf_core_v2]
+    XLSX --> TOOLKIT[aescsf_toolkit_v1_1]
+    XLSX --> GENERIC[generic_xlsx_table]
+    PDF[PDF] --> PDFP[pdf_docling_v1]
+    DOCX[DOCX] --> DOCXP[docx_docling_v1]
+    MD[Markdown] --> MDP[markdown_docling_v1]
+```
+
 - `aescsf_core_v2`
   - Detects the flat AESCSF core workbook layout.
   - Preserves paragraph chunks inside `Context and Guidance`.
@@ -102,3 +179,26 @@ uv run python -m reqif_ingest_cli foundry-config --pretty
 ```
 
 The adapter is for review and remapping only. It is not part of the deterministic first pass.
+
+```mermaid
+flowchart LR
+    DISTILL[Deterministic distillation] --> REVIEW{Foundry configured?}
+    REVIEW -- no --> DONE[Use deterministic output]
+    REVIEW -- yes --> QA[Quality review / remap hints]
+    QA --> DONE
+```
+
+## Current Gaps
+
+- no baseline diff command yet
+- no ingest MCP tool surface yet
+- AESCSF mappings are still code-first rather than externalized config
+- rich PDF structure extraction still depends on offline Docling model availability
+
+```mermaid
+flowchart LR
+    NOW[Current CLI] --> NEXT1[MCP tool surface]
+    NOW --> NEXT2[Baseline diffing]
+    NOW --> NEXT3[Externalized profile config]
+    NOW --> NEXT4[Richer offline PDF structure]
+```