Open
Conversation
Contributor
🔍 Skill Validator Results2 resource(s) checked | ✅ All checks passed Full output |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the eval-driven-dev skill to a newer pixie-qa version and restructures the workflow into smaller, step-scoped reference documents, with helper scripts for setup and background process management.
Changes:
- Bumped skill metadata to
version: 0.4.0and documentedpixie-qaversion/source. - Refactored the skill workflow into Steps 1–7 with per-step reference docs (and added generated API references).
- Added
resources/shell scripts to automate setup, background server start, and server stop; updateddocs/README.skills.mdreferences accordingly.
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| skills/eval-driven-dev/SKILL.md | Updates workflow structure, adds setup/stop guidance, bumps metadata/version. |
| skills/eval-driven-dev/resources/setup.sh | Automates skill update, pixie-qa install/upgrade, pixie init, and starts web UI server. |
| skills/eval-driven-dev/resources/stop-server.sh | Attempts to stop pixie web UI server based on a lock-file/port. |
| skills/eval-driven-dev/resources/run-with-timeout.sh | Utility to run a command in background and kill it after a timeout. |
| skills/eval-driven-dev/references/understanding-app.md | Removes legacy Step 1 reference (replaced by new Step 1a/1b/1c refs). |
| skills/eval-driven-dev/references/run-harness-patterns.md | Removes legacy run harness patterns reference (replaced by new Step 3 + examples). |
| skills/eval-driven-dev/references/pixie-api.md | Removes legacy pixie API reference (replaced by generated API refs). |
| skills/eval-driven-dev/references/instrumentation.md | Removes legacy instrumentation reference (replaced by Step 2 + instrumentation API ref). |
| skills/eval-driven-dev/references/eval-tests.md | Removes legacy eval tests reference (replaced by Step 4–6 + generated refs). |
| skills/eval-driven-dev/references/dataset-generation.md | Removes legacy dataset generation reference (replaced by Step 5 ref). |
| skills/eval-driven-dev/references/1-a-entry-point.md | Adds new Step 1a entry-point artifact guidance. |
| skills/eval-driven-dev/references/1-b-data-flow.md | Adds new Step 1b DAG artifact guidance. |
| skills/eval-driven-dev/references/1-c-eval-criteria.md | Adds new Step 1c eval criteria artifact guidance. |
| skills/eval-driven-dev/references/2-instrument-and-observe.md | Adds new Step 2 instrumentation/trace/DAG validation workflow. |
| skills/eval-driven-dev/references/3-run-harness.md | Adds new Step 3 run harness contract + pointers to examples. |
| skills/eval-driven-dev/references/run-harness-examples/fastapi-web-server.md | Adds FastAPI harness example and concurrency/lifespan notes. |
| skills/eval-driven-dev/references/run-harness-examples/cli-app.md | Adds CLI harness example. |
| skills/eval-driven-dev/references/run-harness-examples/standalone-function.md | Adds standalone-function harness example. |
| skills/eval-driven-dev/references/4-define-evaluators.md | Adds new Step 4 evaluator mapping + custom evaluator guidance. |
| skills/eval-driven-dev/references/evaluators.md | Adds generated built-in evaluator catalog and rules. |
| skills/eval-driven-dev/references/testing-api.md | Adds generated testing API reference and dataset JSON format. |
| skills/eval-driven-dev/references/instrumentation-api.md | Adds generated instrumentation API reference and CLI commands. |
| skills/eval-driven-dev/references/5-build-dataset.md | Adds new Step 5 dataset JSON build/validate guidance. |
| skills/eval-driven-dev/references/6-run-tests.md | Adds new Step 6 test execution + analysis guidance. |
| skills/eval-driven-dev/references/7-investigation.md | Updates investigation workflow to Step 7 and adds a stop/continue gate. |
| docs/README.skills.md | Updates listed reference files/dirs for the skill. |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Checklist
npm startand verified thatREADME.mdis up to date.stagedbranch for this pull request.Description
Update eval-driven-dev skill, with streamlined workflow, and optimized instruction loading with optional reference files that only loads when needed.
Type of Contribution
Additional Notes
By submitting this pull request, I confirm that my contribution abides by the Code of Conduct and will be licensed under the MIT License.