Skip to content

Commit c4124e3

Browse files
committed
Make README more dev-friendly and fix two real bugs in it
- Removed doc/prd.md and doc/task.md from the Documentation table -- they 404 on GitHub since those files were untracked earlier. - Removed the stale "once the repo is public" caveat on the hosted docs line -- the repo is already public. - Added a concrete before/after (flaky assert -> ghostrun) up top so the problem being solved is obvious in the first ten seconds. - Added an honest "Is this for you?" section pointing to DeepEval/ Langfuse/Giskard for use cases ghostrun isn't built for, instead of making every reader discover that by trial and error. - Split the documentation table into "start here" vs "deeper reference" so newcomers aren't handed a 10-row wall of links first.
1 parent 206ab75 commit c4124e3

1 file changed

Lines changed: 42 additions & 9 deletions

File tree

README.md

Lines changed: 42 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -7,12 +7,26 @@
77

88
**pytest for LLMs.** Deterministic record/replay and semantic assertions for GenAI apps — **local-first, privacy-first, zero SaaS lock-in.**
99

10-
Generative AI outputs vary, so `assert output == "expected"` doesn't work. ghostrun gives you two things instead:
10+
### The problem
1111

12-
1. **Deterministic replay** — the first run records real LLM HTTP calls to a local `.ghostrun_cache/`; every run after replays them instantly. Zero API cost, zero latency, zero flakiness.
13-
2. **Semantic assertions** — assert on *meaning* (`contains_intent`, `tone_is`, …), graded by a **local Ollama model** by default. Your prompts and data never leave your machine.
12+
```python
13+
reply = generate_reply("Where is my refund?")
14+
assert reply == "I'm sorry for the delay..." # fails tomorrow: LLM never says the same thing twice
15+
```
16+
17+
Every real test run also means a live API call — slow, costs money, and now your CI needs a secret API key just to run the test suite.
18+
19+
### What ghostrun does about it
1420

15-
No cloud dashboard. No custom CLI to learn. Just `pytest`.
21+
1. **Deterministic replay** — the first run records real LLM HTTP calls to a local `.ghostrun_cache/`; every run after replays them instantly from disk. Zero API cost, zero latency, zero flakiness, no key needed in CI.
22+
2. **Semantic assertions** — assert on *meaning*, not exact text:
23+
```python
24+
ghostrun.expect(reply).contains_intent("apology")
25+
ghostrun.expect(reply).tone_is("empathetic")
26+
```
27+
Graded by a **local Ollama model** by default — your prompts and data never leave your machine. That grading verdict gets cached too, so it's also free and deterministic after the first run.
28+
29+
No cloud dashboard. No custom CLI to learn. No dataset to author by hand. Just `pytest`.
1630

1731
---
1832

@@ -77,25 +91,44 @@ you call a judge-backed assertion (`contains_intent`, `tone_is`, `matches`).
7791
Deterministic assertions (`contains`, `is_valid_json`) and tool-call assertions
7892
never invoke it.
7993

94+
## Is this for you?
95+
96+
**Use ghostrun if** you're writing pytest tests around code that calls an LLM
97+
(directly or via the OpenAI/Anthropic SDKs) and want that suite to run offline,
98+
free, and deterministically after the first recording.
99+
100+
**Skip it if** you need a hosted dashboard/observability platform for
101+
production traffic (see Langfuse/LangSmith/Braintrust instead), you're
102+
building a red-team/adversarial test suite (see Giskard), or you want 50+
103+
pre-built judge metrics out of the box today (see DeepEval — more mature, more
104+
metrics, but doesn't intercept your app's own HTTP calls the way ghostrun
105+
does). See [doc/comparison.md](doc/comparison.md) for the full, researched
106+
breakdown of where ghostrun is ahead and where it's duplicating existing work.
107+
80108
## Documentation
81109

110+
Start here, in order:
111+
82112
| Guide | What's in it |
83113
| :--- | :--- |
84114
| [doc/guide/recording.md](doc/guide/recording.md) | How record/replay works, judge-verdict caching, supported providers, secret redaction, parallel test runs |
85115
| [doc/guide/assertions.md](doc/guide/assertions.md) | Semantic assertions, judge reliability (benchmarked, not asserted), majority-vote verdicts, tool/function-call assertions |
86-
| [doc/guide/regression-tracking.md](doc/guide/regression-tracking.md) | Snapshotting runs, `ghostrun diff`, posting a regression as a PR comment, JUnit CI integration |
87116
| [doc/guide/configuration.md](doc/guide/configuration.md) | `.ghostrun.yaml`, environment variables, pytest flags, `ghostrun doctor`, `ghostrun init` |
117+
118+
Deeper reference, once you're past the basics:
119+
120+
| Guide | What's in it |
121+
| :--- | :--- |
122+
| [doc/guide/regression-tracking.md](doc/guide/regression-tracking.md) | Snapshotting runs, `ghostrun diff`, posting a regression as a PR comment, JUnit CI integration |
88123
| [doc/guide/api-reference.md](doc/guide/api-reference.md) | Every public function, class, exception, and config field |
89124
| [doc/guide/why-not-diy.md](doc/guide/why-not-diy.md) | The actual bugs found building this — the case for a maintained package over a five-minute prompt |
90125
| [doc/judge-voting-benchmark.md](doc/judge-voting-benchmark.md) | Full methodology and results for the majority-vote judge-caching benchmark |
91126
| [doc/comparison.md](doc/comparison.md) | Researched comparison against DeepEval, Promptfoo, Ragas, vcr-langchain, and 9 other tools |
92-
| [doc/prd.md](doc/prd.md) | Product spec |
93-
| [doc/task.md](doc/task.md) | Living status tracker — what's done, what's left, and why |
94127
| [CHANGELOG.md](CHANGELOG.md) | Release notes |
95128

96129
A hosted, searchable version of this documentation is planned at
97-
[parthmax2.github.io/ghostrun](https://parthmax2.github.io/ghostrun/) once the
98-
repo is public (config in `mkdocs.yml`, builds via `.github/workflows/docs.yml`).
130+
[parthmax2.github.io/ghostrun](https://parthmax2.github.io/ghostrun/) (config
131+
in `mkdocs.yml`, builds via `.github/workflows/docs.yml`).
99132

100133
## Contributing
101134

0 commit comments

Comments
 (0)