Skip to content

Commit 71f4029

Browse files
committed
Fix GHOSTRUN_* env var casing left wrong by the gentest rename
The gentest->ghostrun rename turned GENTEST_JUDGE, GENTEST_MODE, etc. into ghostrun_JUDGE, ghostrun_MODE (lowercase prefix) instead of GHOSTRUN_JUDGE, GHOSTRUN_MODE, breaking the documented env var convention across ghostrun/, tests/, examples/, CI, and docs.
1 parent 4aa3004 commit 71f4029

15 files changed

Lines changed: 60 additions & 60 deletions

.github/workflows/ci.yml

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -41,19 +41,19 @@ jobs:
4141
# The core suite is hermetic: no network, no Ollama, no API keys.
4242
- name: Test (offline)
4343
env:
44-
ghostrun_JUDGE: echo
44+
GHOSTRUN_JUDGE: echo
4545
run: pytest tests -q
4646

4747
# The example must skip cleanly rather than fail when no judge exists.
4848
- name: Example degrades gracefully without a judge
4949
env:
50-
ghostrun_JUDGE: echo
50+
GHOSTRUN_JUDGE: echo
5151
run: pytest examples -q
5252

5353
# Replay must work with no network and no judge backend reachable.
5454
- name: Strict replay (no network, judge unreachable)
5555
env:
56-
ghostrun_JUDGE_BASE_URL: http://localhost:59999
56+
GHOSTRUN_JUDGE_BASE_URL: http://localhost:59999
5757
run: pytest examples -q --ghostrun-replay
5858

5959
parallel:
@@ -72,7 +72,7 @@ jobs:
7272
# The interceptor patches global httpx state; this guards against races.
7373
- name: Test under parallel workers
7474
env:
75-
ghostrun_JUDGE: echo
75+
GHOSTRUN_JUDGE: echo
7676
run: pytest tests -q -n 4
7777

7878
build:
@@ -118,6 +118,6 @@ jobs:
118118
- name: Record from the real API, then replay offline
119119
env:
120120
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
121-
ghostrun_LIVE: "1"
122-
ghostrun_JUDGE: echo # don't require Ollama in CI
121+
GHOSTRUN_LIVE: "1"
122+
GHOSTRUN_JUDGE: echo # don't require Ollama in CI
123123
run: pytest examples/test_live_smoke.py -v

CHANGELOG.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ All notable changes to ghostrun are documented here. The format is based on
7373
freezes whatever the judge said on one draw, including if it was wrong —
7474
published LLM-as-judge studies report ~13–14% flip rates on repeated grading
7575
of identical input, even at temperature 0. Setting `judge.votes` (or
76-
`ghostrun_JUDGE_VOTES`) to an odd N > 1 grades N times on a cache miss and
76+
`GHOSTRUN_JUDGE_VOTES`) to an odd N > 1 grades N times on a cache miss and
7777
caches the majority verdict plus the observed disagreement rate
7878
(`Grade.votes`, `Grade.disagreement_rate`). Even vote counts tie-break
7979
conservatively to FAIL. Cache keys now include the vote count, so changing
@@ -104,7 +104,7 @@ All notable changes to ghostrun are documented here. The format is based on
104104
judge could flip a passing test with no code change. Verdicts are now recorded
105105
to `.ghostrun_cache/judge/` under the same `auto`/`record`/`replay` semantics as
106106
HTTP calls, keyed on judge backend + model + text + criterion. Configurable via
107-
`judge.cache` or `ghostrun_JUDGE_CACHE`. Cuts the bundled example from 23.3s to
107+
`judge.cache` or `GHOSTRUN_JUDGE_CACHE`. Cuts the bundled example from 23.3s to
108108
2.7s and makes semantic assertions deterministic.
109109
- Committed tests for async clients and streaming (SSE) record/replay, including
110110
async streaming — previously working but uncovered.
@@ -161,7 +161,7 @@ Initial Phase 1 MVP.
161161
stub for CI.
162162
- **Pytest plugin** (auto-registered): `--ghostrun-record`, `--ghostrun-replay`,
163163
`--ghostrun-judge` flags and a `ghostrun_record` fixture.
164-
- **Configuration** via `.ghostrun.yaml`, `ghostrun_*` environment variables, or
164+
- **Configuration** via `.ghostrun.yaml`, `GHOSTRUN_*` environment variables, or
165165
`ghostrun.configure(...)`, resolved defaults → file → env.
166166
- Documentation (`README.md`, `doc/prd.md`), a runnable `examples/` app with a
167167
pre-recorded cache, and an offline test suite.

CONTRIBUTING.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ real Ollama judge when available and skips gracefully otherwise.
2121

2222
- `pytest tests -q` must pass.
2323
- If you touch the interceptor, cache, or judge-caching logic, also run under
24-
parallel workers: `ghostrun_JUDGE=echo pytest tests -q -n 4` — several real
24+
parallel workers: `GHOSTRUN_JUDGE=echo pytest tests -q -n 4` — several real
2525
bugs in this project (cross-thread cache contamination, torn writes) were
2626
only found this way, not by the sequential suite.
2727
- New behavior needs a test that fails without the fix. This project has

doc/guide/configuration.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@ judge:
1616
cache: true
1717
```
1818
19-
Everything is overridable by environment variable (`ghostrun_MODE`,
20-
`ghostrun_JUDGE`, `ghostrun_JUDGE_MODEL`, `ghostrun_CACHE_DIR`, `ghostrun_JUDGE_VOTES`,
19+
Everything is overridable by environment variable (`GHOSTRUN_MODE`,
20+
`GHOSTRUN_JUDGE`, `GHOSTRUN_JUDGE_MODEL`, `GHOSTRUN_CACHE_DIR`, `GHOSTRUN_JUDGE_VOTES`,
2121
…) or at runtime via `ghostrun.configure(judge="echo")`. Resolution order:
2222
defaults → `.ghostrun.yaml` → environment variables → `ghostrun.configure(...)`.
2323

doc/guide/recording.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ and Anthropic SDKs sit on top of. It does *not* monkey-patch the SDKs, so it
55
survives SDK upgrades. Only requests to known provider hosts are cached; all
66
other traffic passes through untouched.
77

8-
Modes (via `ghostrun_MODE` or `.ghostrun.yaml`):
8+
Modes (via `GHOSTRUN_MODE` or `.ghostrun.yaml`):
99

1010
| Mode | Behavior |
1111
| :--- | :--- |
@@ -39,7 +39,7 @@ test passes:
3939

4040
In the bundled example this takes a run from **23.3s to 2.7s**, and in
4141
`--ghostrun-replay` no model is invoked at all. Disable with `judge.cache: false`
42-
or `ghostrun_JUDGE_CACHE=false` if you want a live grade every run.
42+
or `GHOSTRUN_JUDGE_CACHE=false` if you want a live grade every run.
4343

4444
## Providers
4545

examples/README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ When no real judge is available (echo, or Ollama down / model not pulled), the
3737
example **skips** rather than fails:
3838

3939
```bash
40-
ghostrun_JUDGE=echo pytest examples/test_support_reply.py # -> skipped
40+
GHOSTRUN_JUDGE=echo pytest examples/test_support_reply.py # -> skipped
4141
```
4242

4343
For real semantic grading, use Ollama (above) or point the judge at a cloud
@@ -47,10 +47,10 @@ model in `.ghostrun.yaml`.
4747

4848
[`test_live_smoke.py`](test_live_smoke.py) is the one check the mock-based suite
4949
can't cover: recording from the **real** OpenAI API and replaying it offline.
50-
It's opt-in — skipped unless you provide a key and set `ghostrun_LIVE=1`:
50+
It's opt-in — skipped unless you provide a key and set `GHOSTRUN_LIVE=1`:
5151

5252
```bash
53-
OPENAI_API_KEY=sk-... ghostrun_LIVE=1 pytest examples/test_live_smoke.py -v
53+
OPENAI_API_KEY=sk-... GHOSTRUN_LIVE=1 pytest examples/test_live_smoke.py -v
5454
```
5555

5656
It costs one cheap `gpt-4o-mini` call the first time, writes the response to

examples/test_live_smoke.py

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
44
This is opt-in. It is skipped unless BOTH are true:
55
* OPENAI_API_KEY is set
6-
* ghostrun_LIVE=1
6+
* GHOSTRUN_LIVE=1
77
8-
OPENAI_API_KEY=sk-... ghostrun_LIVE=1 pytest examples/test_live_smoke.py -v
8+
OPENAI_API_KEY=sk-... GHOSTRUN_LIVE=1 pytest examples/test_live_smoke.py -v
99
1010
What it proves, end to end:
1111
1. record -> one real call to api.openai.com is captured to the cache
@@ -29,8 +29,8 @@
2929
from ghostrun.interceptor import CacheMiss, Interceptor
3030

3131
pytestmark = pytest.mark.skipif(
32-
not (os.environ.get("OPENAI_API_KEY") and os.environ.get("ghostrun_LIVE") == "1"),
33-
reason="live test: set OPENAI_API_KEY and ghostrun_LIVE=1 to run",
32+
not (os.environ.get("OPENAI_API_KEY") and os.environ.get("GHOSTRUN_LIVE") == "1"),
33+
reason="live test: set OPENAI_API_KEY and GHOSTRUN_LIVE=1 to run",
3434
)
3535

3636
CACHE_DIR = str(Path(__file__).with_name(".ghostrun_live_cache"))

examples/test_support_reply.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
pytest examples/test_support_reply.py
77
88
The semantic assertions are graded by a local Ollama model (default). If a real
9-
judge isn't available (no Ollama, or ghostrun_JUDGE=echo), the test skips rather
9+
judge isn't available (no Ollama, or GHOSTRUN_JUDGE=echo), the test skips rather
1010
than fails — see examples/conftest.py.
1111
1212
To re-record against the real OpenAI API instead:

ghostrun/config.py

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
Resolution order (lowest to highest precedence):
44
1. Built-in defaults
55
2. ``.ghostrun.yaml`` found by walking up from the current working directory
6-
3. Environment variables (``ghostrun_*``)
6+
3. Environment variables (``GHOSTRUN_*``)
77
88
Nothing here talks to the network; a missing config file is fine.
99
"""
@@ -93,22 +93,22 @@ def _from_file(path: Path) -> dict:
9393
def _from_env() -> dict:
9494
env = os.environ
9595
out: dict = {}
96-
if "ghostrun_MODE" in env:
97-
out["mode"] = env["ghostrun_MODE"]
98-
if "ghostrun_CACHE_DIR" in env:
99-
out["cache_dir"] = env["ghostrun_CACHE_DIR"]
100-
if "ghostrun_JUDGE" in env:
101-
out["judge"] = env["ghostrun_JUDGE"]
102-
if "ghostrun_JUDGE_MODEL" in env:
103-
out["judge_model"] = env["ghostrun_JUDGE_MODEL"]
104-
if "ghostrun_JUDGE_BASE_URL" in env:
105-
out["judge_base_url"] = env["ghostrun_JUDGE_BASE_URL"]
106-
if "ghostrun_JUDGE_TIMEOUT" in env:
107-
out["judge_timeout"] = float(env["ghostrun_JUDGE_TIMEOUT"])
108-
if "ghostrun_JUDGE_CACHE" in env:
109-
out["judge_cache"] = env["ghostrun_JUDGE_CACHE"].strip().lower() not in ("0", "false", "no")
110-
if "ghostrun_JUDGE_VOTES" in env:
111-
out["judge_votes"] = int(env["ghostrun_JUDGE_VOTES"])
96+
if "GHOSTRUN_MODE" in env:
97+
out["mode"] = env["GHOSTRUN_MODE"]
98+
if "GHOSTRUN_CACHE_DIR" in env:
99+
out["cache_dir"] = env["GHOSTRUN_CACHE_DIR"]
100+
if "GHOSTRUN_JUDGE" in env:
101+
out["judge"] = env["GHOSTRUN_JUDGE"]
102+
if "GHOSTRUN_JUDGE_MODEL" in env:
103+
out["judge_model"] = env["GHOSTRUN_JUDGE_MODEL"]
104+
if "GHOSTRUN_JUDGE_BASE_URL" in env:
105+
out["judge_base_url"] = env["GHOSTRUN_JUDGE_BASE_URL"]
106+
if "GHOSTRUN_JUDGE_TIMEOUT" in env:
107+
out["judge_timeout"] = float(env["GHOSTRUN_JUDGE_TIMEOUT"])
108+
if "GHOSTRUN_JUDGE_CACHE" in env:
109+
out["judge_cache"] = env["GHOSTRUN_JUDGE_CACHE"].strip().lower() not in ("0", "false", "no")
110+
if "GHOSTRUN_JUDGE_VOTES" in env:
111+
out["judge_votes"] = int(env["GHOSTRUN_JUDGE_VOTES"])
112112
return out
113113

114114

ghostrun/interceptor.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ def handle_request(self, request: httpx.Request) -> httpx.Response:
7272
if self._mode == "replay":
7373
raise CacheMiss(
7474
f"No cached response for {request.method} {request.url} "
75-
f"(key {key}). Re-run with ghostrun_MODE=record to capture it."
75+
f"(key {key}). Re-run with GHOSTRUN_MODE=record to capture it."
7676
)
7777

7878
response = self._inner.handle_request(request)
@@ -102,7 +102,7 @@ async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
102102
if self._mode == "replay":
103103
raise CacheMiss(
104104
f"No cached response for {request.method} {request.url} "
105-
f"(key {key}). Re-run with ghostrun_MODE=record to capture it."
105+
f"(key {key}). Re-run with GHOSTRUN_MODE=record to capture it."
106106
)
107107

108108
response = await self._inner.handle_async_request(request)

0 commit comments

Comments
 (0)