Add README and enhance developer guide with custom prompt, schema, and evaluation details

Author: aaronkaplan

README.md

@@ -0,0 +1,62 @@
+# LLM Batch Pipeline
+
+`llm-batch-pipeline` is a generic LLM batch processing pipeline. It discovers and parses input files via a plugin system, renders OpenAI Batch API (or Ollama) requests, validates structured JSON outputs with a Pydantic schema, evaluates predictions against ground truth, and exports results to XLSX/JSON.
+
+See the **Admin Guide** for installation/deployment, and the **Developer Guide** for how to extend the pipeline with custom plugins, prompts, schemas, and evaluation.
+
+## Workflow
+```mermaid
+flowchart TD
+    A[Input Files] --> B[1. Discover]
+    B --> C[2. Filter - pre]
+    C --> D[3. Transform]
+    D --> E[4. Filter - post]
+    E --> F[5. Render JSONL]
+    F --> G{6. Human Review}
+    G -->|Approved| H[7. Submit to Backend]
+    G -->|--auto-approve| H
+    G -->|Rejected| Z[Abort]
+    H --> I[8. Validate Results]
+    I --> J[9. Evaluate]
+    J --> K[10. Export]
+
+    subgraph Backends
+        H --> H1[OpenAI Batch API]
+        H --> H2[Ollama Local]
+    end
+
+    H1 --> I
+    H2 --> I
+
+    subgraph Outputs
+        K --> K1[results.xlsx]
+        K --> K2[evaluation.xlsx]
+        K --> K3[evaluation.json]
+        K --> K4[metrics.json]
+    end
+```
+
+## Admin / Install
+- Admin guide: [`docs/admin-guide.md`](docs/admin-guide.md)
+- Install (example): `uv sync`
+
+## Requirements
+- OpenAI backend: set `OPENAI_API_KEY`.
+- Local LLM via Ollama: run an Ollama server (pull the model), then use `--backend ollama --base-url http://HOST:11434` (repeat `--base-url` for multi-server sharding).
+- OpenAI API compatible local server (if supported by your server): use `--backend openai` and configure the OpenAI SDK base URL (commonly via `OPENAI_BASE_URL`).
+
+## Test / Benchmark
+- SpamAssassin corpus reference run: [`docs/benchmark-run.md`](docs/benchmark-run.md)
+
+## Architecture
+- [`docs/architecture.md`](docs/architecture.md)
+
+## Extend (plugins)
+- [`docs/developer-guide.md`](docs/developer-guide.md)
+- Developer guide covers: custom prompt, custom Pydantic schema, and custom evaluation.
+
+## Monitor
+- Prometheus metrics + Grafana: [`docs/admin-guide.md`](docs/admin-guide.md)
+
+## License
+- EUPL (v1.2): [see `LICENSE`](LICENSE)

docs/developer-guide.md

@@ -109,7 +109,7 @@ class SentimentResult(BaseModel):
     )
     reason: str = Field(description="Explanation.")
 
-# REQUIRED: the pipeline loads schemas by this name
+# REQUIRED: `schema_loader` expects a symbol named `mySchema`
 mySchema = SentimentResult
 ```
 
@@ -126,7 +126,6 @@ def register():
         pre_filters=[MinRowFilter(min_rows=2)],
         transformers=[NormalizeHeadersTransformer()],
         post_filters=[],
-        default_schema_module="my_plugin.schema",
     ))
 ```
 
@@ -149,6 +148,55 @@ from my_plugin.plugin import register
 register()
 ```
 
+## Custom Prompt, Schema, and Evaluation (per batch)
+
+The plugin system controls how inputs are parsed and preprocessed. Prompt/instructions, schema validation, and evaluation configuration are provided per *batch run*.
+
+### Custom prompt (instructions)
+
+The LLM `instructions` text is loaded from:
+
+- `--prompt-file` (CLI) or `batches/<batch>/prompt.txt` (file in the batch directory)
+- a built-in fallback prompt when neither exists
+
+For submit-time overrides, use `--prompt-override` / `--prompt-override-file`.
+The backends apply this by rewriting `body.instructions` for every request right before submission.
+
+Your plugin’s `FileReader.package_for_llm()` controls the per-file content injected as the `input_text` part of the prompt.
+
+### Custom Pydantic schema
+
+Provide a schema via one of these mechanisms:
+
+- `--schema-file path/to/schema.py`
+- or a `schema.py` file placed in the batch directory (`batches/<batch>/schema.py`)
+
+Your schema file must define `mySchema` (typically `class mySchema(BaseModel): ...` or `mySchema = SomeModel`).
+The pipeline converts it to a strict JSON schema for structured outputs and then validates the LLM JSON against it.
+
+Evaluation field mapping defaults to `label` and `confidence`. To evaluate other schema field names, use `--label-field` and `--confidence-field`.
+
+Note: evaluation auto-detection of label/confidence from the schema runs only when `--schema-file` is explicitly provided.
+If you rely solely on `batches/<batch>/schema.py`, set `--label-field` / `--confidence-field` explicitly.
+
+### Custom evaluation
+
+The built-in `evaluate` stage computes confusion matrix + precision/recall/F1 + accuracy, and optionally ROC/AUC for binary classification when `--positive-class` is set (and confidence is available).
+
+It uses:
+
+- Ground truth from `--ground-truth-csv` or `batches/<batch>/evaluation/ground-truth.csv`
+- Category map from `--category-map` or `batches/<batch>/evaluation/category-map.json`
+
+Supported 'custom evaluation' via plugins:
+
+- Implement `OutputTransformer` in your plugin to reshape `validated_rows` before evaluation/export (e.g. rename/move fields so they match `--label-field` / `--confidence-field`).
+
+If you need entirely different evaluation metrics/logic:
+
+- modify `src/llm_batch_pipeline/evaluation.py` (`evaluate()` / `EvalReport`)
+- and update `src/llm_batch_pipeline/stages.py` `stage_evaluate()` (plus `src/llm_batch_pipeline/export.py` if the report format changes)
+
 ## Plugin ABCs Reference
 
 ### `FileReader`