|
| 1 | +# Extraction submodule |
| 2 | + |
| 3 | +This folder contains a **file-driven, multi-pass extraction pipeline** built on **Outlines** (JSON-guided generation) and **Ollama** (local model serving). The pipeline reads configuration, prompts, and JSON Schemas from disk, runs a configured sequence of passes, writes versioned artifacts to an output directory (never overwriting), and can optionally persist results + timing metadata into an SQLite database. |
| 4 | + |
| 5 | +## What it does |
| 6 | + |
| 7 | +- **Pass-based extraction**: runs a configurable sequence of passes (e.g., `A_core`, `B_index`, `C_sequences`, ...) using Outlines JSON schema guidance. |
| 8 | +- **Artifacts on disk**: writes raw text, pretty JSON, and error logs for each pass without overwriting prior runs. |
| 9 | +- **Final stitching + validation**: stitches per-pass outputs into a final “FULL” object and can validate it against a “full schema” if configured. |
| 10 | +- **SQLite optional**: can insert stitched results into SQLite via `hyb_db.insert_article_object(...)`. |
| 11 | +- **Perf sidecars + continuation**: each JSON artifact can have a `*.perf.json` sidecar; the same metrics can also be mirrored into SQLite (`pipeline_artifacts`) and used for “resume” mode. |
| 12 | + |
| 13 | +## Repository layout |
| 14 | + |
| 15 | +The pipeline expects a “project directory” that contains: |
| 16 | + |
| 17 | +- `config/pipeline.json` (main configuration) |
| 18 | +- `passes/<pass_name>/schema.json` and `passes/<pass_name>/prompt.txt` (per-pass assets) |
| 19 | +- `passes/common.txt` (shared prompt prefix, optional) |
| 20 | +- `schema/json/article.json` (full schema for final validation, optional) |
| 21 | +- input directory with source files (configured in `pipeline.json`) |
| 22 | + |
| 23 | +The config shown in `config/pipeline.json` includes keys such as: |
| 24 | +- `model_names`, `ollama_base_url`, `ollama_parameters`, `timeout_s` |
| 25 | +- `input_dir`, `out_dir`, `article_glob` |
| 26 | +- `pre_passes`, `construct_single_experiment_passes`, `passes` |
| 27 | + |
| 28 | +## Installation |
| 29 | + |
| 30 | +Python dependencies (minimum set used by the pipeline): |
| 31 | + |
| 32 | +```bash |
| 33 | +pip install -r requirements.txt |
| 34 | +``` |
| 35 | + |
| 36 | +Or use the conda/mamba to initialize environment from `environment.yml`. |
| 37 | + |
| 38 | +You also need: |
| 39 | +- **Ollama** running locally (or reachable over HTTP), matching `ollama_base_url` in `config/pipeline.json`. |
| 40 | + |
| 41 | +Optional: |
| 42 | +- If `db_path` is set in config, SQLite will be used and schema will be auto-created. |
| 43 | + |
| 44 | +### Environment variables |
| 45 | + |
| 46 | +- `OPEN_BUTTON_TOKEN` (optional): if set, it is passed as a Bearer token in Ollama client headers. |
| 47 | + |
| 48 | +## How to run |
| 49 | + |
| 50 | +### 1) Configure `config/pipeline.json` |
| 51 | + |
| 52 | +Edit paths to match your machine. In the attached example, `input_dir` is set to an absolute path and `article_glob` uses a recursive pattern. |
| 53 | + |
| 54 | +Key fields you typically tune: |
| 55 | +- `model_names`: list of Ollama model identifiers to run. |
| 56 | +- `ollama_parameters`: e.g. `num_ctx`, `num_predict`, `temperature`, `seed`. |
| 57 | +- `timeout_s`, `ollama_base_url` |
| 58 | +- `out_dir`, `db_path` |
| 59 | + |
| 60 | +### 2) Run the pipeline |
| 61 | + |
| 62 | +#### CLI |
| 63 | + |
| 64 | +From the repository root (or anywhere, as long as you pass the correct project directory): |
| 65 | + |
| 66 | +```bash |
| 67 | +python extraction/pipeline_filedriven.py extraction --fresh |
| 68 | +``` |
| 69 | + |
| 70 | +- `project_dir` is the folder containing `config/`, `passes/`, etc. |
| 71 | +- omit `--fresh` to enable continuation/resume behavior. |
| 72 | + |
| 73 | +#### Python |
| 74 | + |
| 75 | +```python |
| 76 | +from extraction.pipeline_filedriven import run_project |
| 77 | +run_project("extraction", fresh=False) |
| 78 | +``` |
| 79 | + |
| 80 | +## Outputs |
| 81 | + |
| 82 | +Artifacts are written under `out_dir` (from `pipeline.json`). |
| 83 | + |
| 84 | +The pipeline writes, per pass and per model/article: |
| 85 | +- raw text: `*.txt` |
| 86 | +- JSON outputs: `*.json` |
| 87 | +- log JSON: `*.log.json` |
| 88 | +- error logs: `logs/*.log` |
| 89 | +- perf sidecars: `*.perf.json` (one per emitted JSON artifact) |
| 90 | + |
| 91 | +Perf sidecars include timestamps, wallclock duration, and (when Ollama reports it) token counts. |
| 92 | + |
| 93 | +## Continuation / resume mode |
| 94 | + |
| 95 | +When `db_path` is configured, the pipeline can skip already completed work: |
| 96 | + |
| 97 | +- default `fresh=False`: for each `(model_name, article_name)`, if a successful `pass_name="FULL"` is recorded in the DB, the article can be skipped. |
| 98 | +- `--fresh`: disables skipping and forces re-processing. |
| 99 | + |
| 100 | +Implementation note: |
| 101 | +- completion is tracked in `pipeline_artifacts` and queried via `hyb_db.get_completed_passes(...)`. |
| 102 | + |
| 103 | +## Database schema (optional) |
| 104 | + |
| 105 | +If `db_path` is set, `hyb_db` auto-creates tables and views and inserts: |
| 106 | +- stitched article objects (`insert_article_object`) |
| 107 | +- artifact-level perf bookkeeping (`pipeline_artifacts`) |
| 108 | + |
| 109 | +## Overall design (short) |
| 110 | + |
| 111 | +- **Config-first**: a project is a directory of config + prompts + schemas, making experiments easy to reproduce and version-control. |
| 112 | +- **Multi-pass extraction**: each pass targets a specific sub-problem and produces a structured JSON artifact. |
| 113 | +- **Immutable artifacts**: outputs are timestamped and never overwritten, enabling auditing and comparisons across runs. |
| 114 | +- **Optional persistence**: results and metrics can be stored in SQLite for analysis and “resume” behavior. |
0 commit comments