evalci

Statistically rigorous LLM evaluation. A single accuracy number with no confidence interval hides whether a change is real or noise. evalci takes per-sample eval scores and reports means with confidence intervals (CLT and bootstrap), handles the non-iid case with clustered standard errors, compares two runs with a paired analysis, computes the sample size needed to detect an effect, and corrects for a noisy LLM judge.

The headline command answers the question every eval pipeline should ask before shipping a model or prompt change:

evalci is-this-regression-real baseline.jsonl candidate.jsonl

It returns REGRESSION, IMPROVEMENT, or NO SIGNIFICANT DIFFERENCE, with an exit code suitable for a CI gate.

evalci is provider-agnostic: it operates on score files, so it works with any model or harness. A fast inference backend (this repo uses Groq) makes generating the large sample sizes needed for tight intervals, and the thousands of bootstrap resamples behind them, cheap.

Why

Reporting one eval number with no interval is a tell that the statistics haven't been done. Two runs differ by 2 points — is that a real regression, or would it vanish on a re-run? Without a confidence interval you cannot tell, and a CI gate built on the raw number will flap on noise. evalci makes the uncertainty explicit and the decision defensible.

What it does

Confidence intervals, done correctly

evalci summarize run.jsonl --method bootstrap
# mean=0.7250  95% CI=[0.6664, 0.7836]  SEM=0.0280  n=200  (clustered)

• CLT interval — fast normal-approximation, the default.
• Bootstrap interval — percentile bootstrap of the mean, no distributional assumption (10,000 resamples by default), better for small n or skewed metrics.
• Clustered standard errors — when samples share structure (e.g. several questions per source document), the iid SE is too small. If the input has a cluster field, evalci uses a cluster-robust estimator automatically, and the interval correctly widens.

Paired A-vs-B significance

evalci is-this-regression-real baseline.jsonl candidate.jsonl

A mean: 0.7250
B mean: 0.6550
difference (B - A): -0.0700
95% CI (bootstrap): [-0.1050, -0.0350]
95% CI (paired CLT): [-0.1057, -0.0343]
p-value (paired): 1e-04
verdict: REGRESSION
(paired on 200 common items)

Runs scored on the same items are compared paired (per-item differences), which removes per-item difficulty from the variance and is far more powerful than comparing two independent means. Both a paired CLT test and a non-parametric paired bootstrap are run; when they disagree on significance, evalci says so (a sign the normal approximation is unreliable at that n).

Exit codes: 1 on a significant regression, 0 otherwise — drop it into CI.

Power analysis

evalci power baseline.jsonl --effect 0.02
# To detect an effect of 0.02 at 80% power, you need n ≈ 3932 samples.

When a comparison is NO SIGNIFICANT DIFFERENCE, the honest question is "zero effect, or underpowered?" Power analysis gives the quantitative answer: the minimum detectable effect at your current n, or the n required for a target effect.

Noisy-judge correction

An LLM judge with a known error rate biases the measured pass rate. Calibrate the judge against a small gold set, then de-bias the headline number with the Rogan-Gladen estimator (evalci.judge).

Correctness

The statistics are validated, not just asserted. The test suite includes simulation-based coverage checks — generate many datasets from a known distribution and verify that a 95% CI actually contains the true mean ~95% of the time, and that the comparison's false-positive rate stays near 5% when two runs are drawn from the same distribution. These are the checks that catch a subtly wrong interval.

Installation

git clone https://github.com/qazasd2518995/evalci.git
cd evalci

python -m venv venv && source venv/bin/activate
pip install -e .

No API key is required to use evalci itself.

Input format

One JSON object per line; score is required, cluster is optional.

{"id": "q1", "score": 1, "cluster": "doc-A"}
{"id": "q2", "score": 0, "cluster": "doc-A"}

Comparisons pair the two files by id.

Library usage

from evalci import summarize, compare, sample_size_for

s = summarize(scores, method="bootstrap")
print(s.mean, s.ci_low, s.ci_high)

cmp = compare(baseline_scores, candidate_scores)   # paired
print(cmp.verdict, cmp.explain())

n = sample_size_for(effect=0.015, sd_diff=0.45)     # samples to detect 1.5pp

Use it as a CI gate

- name: Gate on eval regression
  run: evalci is-this-regression-real examples/baseline.jsonl examples/candidate.jsonl

The step fails the build only when the candidate is a statistically significant regression — not when it dips by noise.

Testing

pip install -e . pytest
pytest -q     # 18 tests, including coverage and false-positive-rate simulations

Project layout

evalci/
  stats.py    means, CLT/bootstrap CIs, clustered standard errors
  compare.py  paired A-vs-B: CLT + bootstrap, verdict, agreement check
  power.py    minimum detectable effect / required sample size
  judge.py    Rogan-Gladen noisy-judge correction
  io.py       per-sample JSONL loading + id-based pairing
  cli.py      summarize / is-this-regression-real / power
tests/        statistical correctness + simulation-based coverage checks
examples/     a baseline/candidate pair to try the gate on

License

MIT