feat: add eval-schemas, eval-fixtures, eval-notebook checks#127
Open
adnanrhussain wants to merge 3 commits into
Open
feat: add eval-schemas, eval-fixtures, eval-notebook checks#127adnanrhussain wants to merge 3 commits into
adnanrhussain wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds three new evaluator artifact validation checks to the existing scripts/check.py harness, rounding out schema/fixture/notebook consistency checks introduced in the prior stacked PR.
Changes:
- Adds
eval-schemasto meta-validateinput_schema.json/output_schema.jsonas JSON Schema documents. - Adds
eval-fixturesto validatefixtures.jsonstructure and bind fixtureinput/expectedto the evaluator’s schemas. - Adds
eval-notebookto heuristically verify example notebooks loadconfig.jsonand prompt files from disk, plus updates documentation/registry.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/README.md | Documents the new checks in the harness README. |
| scripts/checks/eval_schemas.py | New check for JSON Schema meta-validation of evaluator schemas. |
| scripts/checks/eval_fixtures.py | New check for fixtures schema validation + per-evaluator binding. |
| scripts/checks/eval_notebook.py | New check ensuring notebooks reference config/prompts on disk (heuristic). |
| scripts/checks/base.py | Simplifies shared_schema_path() helper to a single-argument form. |
| scripts/checks/init.py | Registers the new checks in the harness. |
| README.md | Links to scripts/ harness and recommends running scripts/check.py --fix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
9c396b6 to
5db7191
Compare
…e tracked_files for notebooks
czi-fsisenda
approved these changes
Jun 30, 2026
czi-fsisenda
left a comment
Contributor
There was a problem hiding this comment.
Nice! Great way to enforce correct use of the json patterns for the notebooks.
If we're including fixtures jsons, should we find a way to converge the json pattern with the contract tests?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Three new checks, completing the evaluator-artifact validation set started in #120:
eval-schemasinput_schema.json/output_schema.jsonis a well-formed JSON Schema document (meta-validation)eval-fixturesfixtures.jsonagainst the sharedevals/_schemas/fixtures.schema.json, and binds each case'sinputto the evaluator'sinput_schemaand eachexpectedfield to itsoutput_schemapropertyeval-notebookconfig.json+ the declared prompt files from disk rather than hardcoding themAll register into the same harness; CI runs them via
python scripts/check.py.Notes
eval-fixturesdegrades gracefully if an evaluator's schema is itself malformed (that'seval-schemas' job to report) — guarded withcheck_schemaso it never crashes the run.eval-notebookis a lightweight heuristic (string-level), not notebook execution.