Skip to content

llm-eval-quality-gates.md

Latest commit

 

History

History
72 lines (51 loc) · 3.86 KB

llm-eval-quality-gates.md

File metadata and controls

72 lines (51 loc) · 3.86 KB
Raw

LLM eval quality gates

The quality gate is the release decision: a skill package ships only when its verification metrics clear the thresholds the requirement itself declares. Implementation: src/core/quality-gate.ts.

The four metrics

All four are computed per verification run over the eval case results:

Metric Definition (local-demo verifier) Gate rule
passRate cases whose checks all pass and whose expected status is declared / total cases >= min_pass_rate
schemaValidRate cases whose expected status is part of the declared output contract / total >= min_schema_valid_rate
unsupportedClaimRate cases flagged by a failed no_unsupported_claims check / total <= max_unsupported_claim_rate
toolErrorRate failed tool_failure-category cases / total <= max_tool_error_rate

On top of the metric rules, any failed static check fails the gate — a missing file, invalid manifest, or unconfirmed write tool is unreleasable regardless of pass rate. Each violation becomes a human-readable reason string in the gate result:

{
  "passed": false,
  "reasons": [
    "pass rate 27.3% is below minimum 90.0%",
    "schema valid rate 90.9% is below minimum 100.0%",
    "static check failed: write_tools_confirmation (All write tools require explicit user confirmation)"
  ],
  "metrics": { "passRate": 0.273, "schemaValidRate": 0.909, "unsupportedClaimRate": 0.364, "toolErrorRate": 0.182 }
}

Thresholds live in the requirement

The skill requirement YAML owns its quality bar:

quality_gate:
  min_pass_rate: 0.90
  min_schema_valid_rate: 1.00
  max_unsupported_claim_rate: 0.02
  max_tool_error_rate: 0.05

Omit the block and the defaults above apply (src/config/default-config.ts). Guidance: keep min_schema_valid_rate at 1.00 — a skill that can emit undeclared statuses breaks every consumer; loosen min_pass_rate only for exploratory skills, and say why in the requirement file, which is versioned in git.

What a failed gate does

  • package_if_passed records a failed stage with the gate reasons; no zip is written to the run dir or dist/.
  • The terminal summary prints Result: FAILED plus each reason; the HTML report shows thresholds vs. actuals side by side.
  • The CLI exits 1 (gate failed) vs. 0 (passed) vs. 2 (configuration/pipeline error) — pipe-friendly for CI.

--allow-failed-package overrides the block for debugging. It is labeled UNSAFE, the stage note says the gate failed, and the run still reports FAILED — the override produces an artifact, never a green build.

Gates cannot be gamed from inside the loop

Two structural guarantees keep the gate honest:

  1. Repairs may not touch the tests. src/repair/skill-repairer.ts rejects any adapter output that modifies eval-cases.json or verification-rules.json and logs the rejection into the repair changes.
  2. Thresholds are read from the requirement, not from the repaired package. A repair that edited the manifest's quality gate would change nothing — the gate evaluates against the spec's thresholds.

The only legitimate ways to pass are improving the skill or consciously editing the requirement (which is a reviewable git diff).

Using the gate in CI

The repo's own workflow (.github/workflows/ci.yml) is the reference pattern:

- run: npm run forge
- run: |
    test -f runs/latest/reports/summary.json
    node -e "const s=require('./runs/latest/reports/summary.json'); if(!s.passed) process.exit(1)"

Because the default model and verifier are offline and deterministic, the gate produces the same verdict on every machine — no flaky API calls in the release path. When real model adapters land (roadmap.md), the recommended pattern is unchanged: generation may be stochastic, but the gate stays deterministic over whatever was generated.