|
7 | 7 |
|
8 | 8 | **pytest for LLMs.** Deterministic record/replay and semantic assertions for GenAI apps — **local-first, privacy-first, zero SaaS lock-in.** |
9 | 9 |
|
10 | | -Generative AI outputs vary, so `assert output == "expected"` doesn't work. ghostrun gives you two things instead: |
| 10 | +### The problem |
11 | 11 |
|
12 | | -1. **Deterministic replay** — the first run records real LLM HTTP calls to a local `.ghostrun_cache/`; every run after replays them instantly. Zero API cost, zero latency, zero flakiness. |
13 | | -2. **Semantic assertions** — assert on *meaning* (`contains_intent`, `tone_is`, …), graded by a **local Ollama model** by default. Your prompts and data never leave your machine. |
| 12 | +```python |
| 13 | +reply = generate_reply("Where is my refund?") |
| 14 | +assert reply == "I'm sorry for the delay..." # fails tomorrow: LLM never says the same thing twice |
| 15 | +``` |
| 16 | + |
| 17 | +Every real test run also means a live API call — slow, costs money, and now your CI needs a secret API key just to run the test suite. |
| 18 | + |
| 19 | +### What ghostrun does about it |
14 | 20 |
|
15 | | -No cloud dashboard. No custom CLI to learn. Just `pytest`. |
| 21 | +1. **Deterministic replay** — the first run records real LLM HTTP calls to a local `.ghostrun_cache/`; every run after replays them instantly from disk. Zero API cost, zero latency, zero flakiness, no key needed in CI. |
| 22 | +2. **Semantic assertions** — assert on *meaning*, not exact text: |
| 23 | + ```python |
| 24 | + ghostrun.expect(reply).contains_intent("apology") |
| 25 | + ghostrun.expect(reply).tone_is("empathetic") |
| 26 | + ``` |
| 27 | + Graded by a **local Ollama model** by default — your prompts and data never leave your machine. That grading verdict gets cached too, so it's also free and deterministic after the first run. |
| 28 | + |
| 29 | +No cloud dashboard. No custom CLI to learn. No dataset to author by hand. Just `pytest`. |
16 | 30 |
|
17 | 31 | --- |
18 | 32 |
|
@@ -77,25 +91,44 @@ you call a judge-backed assertion (`contains_intent`, `tone_is`, `matches`). |
77 | 91 | Deterministic assertions (`contains`, `is_valid_json`) and tool-call assertions |
78 | 92 | never invoke it. |
79 | 93 |
|
| 94 | +## Is this for you? |
| 95 | + |
| 96 | +**Use ghostrun if** you're writing pytest tests around code that calls an LLM |
| 97 | +(directly or via the OpenAI/Anthropic SDKs) and want that suite to run offline, |
| 98 | +free, and deterministically after the first recording. |
| 99 | + |
| 100 | +**Skip it if** you need a hosted dashboard/observability platform for |
| 101 | +production traffic (see Langfuse/LangSmith/Braintrust instead), you're |
| 102 | +building a red-team/adversarial test suite (see Giskard), or you want 50+ |
| 103 | +pre-built judge metrics out of the box today (see DeepEval — more mature, more |
| 104 | +metrics, but doesn't intercept your app's own HTTP calls the way ghostrun |
| 105 | +does). See [doc/comparison.md](doc/comparison.md) for the full, researched |
| 106 | +breakdown of where ghostrun is ahead and where it's duplicating existing work. |
| 107 | + |
80 | 108 | ## Documentation |
81 | 109 |
|
| 110 | +Start here, in order: |
| 111 | + |
82 | 112 | | Guide | What's in it | |
83 | 113 | | :--- | :--- | |
84 | 114 | | [doc/guide/recording.md](doc/guide/recording.md) | How record/replay works, judge-verdict caching, supported providers, secret redaction, parallel test runs | |
85 | 115 | | [doc/guide/assertions.md](doc/guide/assertions.md) | Semantic assertions, judge reliability (benchmarked, not asserted), majority-vote verdicts, tool/function-call assertions | |
86 | | -| [doc/guide/regression-tracking.md](doc/guide/regression-tracking.md) | Snapshotting runs, `ghostrun diff`, posting a regression as a PR comment, JUnit CI integration | |
87 | 116 | | [doc/guide/configuration.md](doc/guide/configuration.md) | `.ghostrun.yaml`, environment variables, pytest flags, `ghostrun doctor`, `ghostrun init` | |
| 117 | + |
| 118 | +Deeper reference, once you're past the basics: |
| 119 | + |
| 120 | +| Guide | What's in it | |
| 121 | +| :--- | :--- | |
| 122 | +| [doc/guide/regression-tracking.md](doc/guide/regression-tracking.md) | Snapshotting runs, `ghostrun diff`, posting a regression as a PR comment, JUnit CI integration | |
88 | 123 | | [doc/guide/api-reference.md](doc/guide/api-reference.md) | Every public function, class, exception, and config field | |
89 | 124 | | [doc/guide/why-not-diy.md](doc/guide/why-not-diy.md) | The actual bugs found building this — the case for a maintained package over a five-minute prompt | |
90 | 125 | | [doc/judge-voting-benchmark.md](doc/judge-voting-benchmark.md) | Full methodology and results for the majority-vote judge-caching benchmark | |
91 | 126 | | [doc/comparison.md](doc/comparison.md) | Researched comparison against DeepEval, Promptfoo, Ragas, vcr-langchain, and 9 other tools | |
92 | | -| [doc/prd.md](doc/prd.md) | Product spec | |
93 | | -| [doc/task.md](doc/task.md) | Living status tracker — what's done, what's left, and why | |
94 | 127 | | [CHANGELOG.md](CHANGELOG.md) | Release notes | |
95 | 128 |
|
96 | 129 | A hosted, searchable version of this documentation is planned at |
97 | | -[parthmax2.github.io/ghostrun](https://parthmax2.github.io/ghostrun/) once the |
98 | | -repo is public (config in `mkdocs.yml`, builds via `.github/workflows/docs.yml`). |
| 130 | +[parthmax2.github.io/ghostrun](https://parthmax2.github.io/ghostrun/) (config |
| 131 | +in `mkdocs.yml`, builds via `.github/workflows/docs.yml`). |
99 | 132 |
|
100 | 133 | ## Contributing |
101 | 134 |
|
|
0 commit comments