Skip to content

feat: add eval-schemas, eval-fixtures, eval-notebook checks#127

Open
adnanrhussain wants to merge 3 commits into
ahussain/eval-config-checksfrom
ahussain/eval-artifact-checks
Open

feat: add eval-schemas, eval-fixtures, eval-notebook checks#127
adnanrhussain wants to merge 3 commits into
ahussain/eval-config-checksfrom
ahussain/eval-artifact-checks

Conversation

@adnanrhussain

@adnanrhussain adnanrhussain commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Stacked on #120 (ahussain/eval-config-checks). Review/merge that first; this PR's diff is against that base.

What

Three new checks, completing the evaluator-artifact validation set started in #120:

Check Validates
eval-schemas each input_schema.json / output_schema.json is a well-formed JSON Schema document (meta-validation)
eval-fixtures fixtures.json against the shared evals/_schemas/fixtures.schema.json, and binds each case's input to the evaluator's input_schema and each expected field to its output_schema property
eval-notebook where an evaluator ships an example notebook, confirms its code loads config.json + the declared prompt files from disk rather than hardcoding them

All register into the same harness; CI runs them via python scripts/check.py.

Notes

  • eval-fixtures degrades gracefully if an evaluator's schema is itself malformed (that's eval-schemas' job to report) — guarded with check_schema so it never crashes the run.
  • eval-notebook is a lightweight heuristic (string-level), not notebook execution.
  • All 8 current evaluators pass all five checks. Negative-tested each new check to confirm it actually fires.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds three new evaluator artifact validation checks to the existing scripts/check.py harness, rounding out schema/fixture/notebook consistency checks introduced in the prior stacked PR.

Changes:

  • Adds eval-schemas to meta-validate input_schema.json / output_schema.json as JSON Schema documents.
  • Adds eval-fixtures to validate fixtures.json structure and bind fixture input/expected to the evaluator’s schemas.
  • Adds eval-notebook to heuristically verify example notebooks load config.json and prompt files from disk, plus updates documentation/registry.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
scripts/README.md Documents the new checks in the harness README.
scripts/checks/eval_schemas.py New check for JSON Schema meta-validation of evaluator schemas.
scripts/checks/eval_fixtures.py New check for fixtures schema validation + per-evaluator binding.
scripts/checks/eval_notebook.py New check ensuring notebooks reference config/prompts on disk (heuristic).
scripts/checks/base.py Simplifies shared_schema_path() helper to a single-argument form.
scripts/checks/init.py Registers the new checks in the harness.
README.md Links to scripts/ harness and recommends running scripts/check.py --fix.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread scripts/checks/eval_fixtures.py
Comment thread scripts/checks/eval_fixtures.py
Comment thread scripts/checks/eval_fixtures.py
Comment thread scripts/checks/eval_notebook.py Outdated
Comment thread scripts/checks/eval_notebook.py Outdated
@adnanrhussain adnanrhussain force-pushed the ahussain/eval-artifact-checks branch from 9c396b6 to 5db7191 Compare June 29, 2026 19:51

@czi-fsisenda czi-fsisenda left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Great way to enforce correct use of the json patterns for the notebooks.
If we're including fixtures jsons, should we find a way to converge the json pattern with the contract tests?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants