Skip to content

update eval-driven-dev skill#1306

Open
yiouli wants to merge 2 commits intogithub:stagedfrom
yiouli:staged
Open

update eval-driven-dev skill#1306
yiouli wants to merge 2 commits intogithub:stagedfrom
yiouli:staged

Conversation

@yiouli
Copy link
Copy Markdown
Contributor

@yiouli yiouli commented Apr 5, 2026

Pull Request Checklist

  • I have read and followed the CONTRIBUTING.md guidelines.
  • I have read and followed the Guidance for submissions involving paid services.
  • My contribution adds a new instruction, prompt, agent, skill, or workflow file in the correct directory.
  • The file follows the required naming convention.
  • The content is clearly structured and follows the example format.
  • I have tested my instructions, prompt, agent, skill, or workflow with GitHub Copilot.
  • I have run npm start and verified that README.md is up to date.
  • I am targeting the staged branch for this pull request.

Description

Update eval-driven-dev skill, with streamlined workflow, and optimized instruction loading with optional reference files that only loads when needed.


Type of Contribution

  • New instruction file.
  • New prompt file.
  • New agent file.
  • New plugin.
  • New skill file.
  • New agentic workflow.
  • Update to existing instruction, prompt, agent, plugin, skill, or workflow.
  • Other (please specify):

Additional Notes


By submitting this pull request, I confirm that my contribution abides by the Code of Conduct and will be licensed under the MIT License.

Copilot AI review requested due to automatic review settings April 5, 2026 19:21
@yiouli yiouli requested a review from aaronpowell as a code owner April 5, 2026 19:21
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 5, 2026

🔍 Skill Validator Results

2 resource(s) checked | ✅ All checks passed

Full output
Found 1 skill(s)
[eval-driven-dev] 📊 eval-driven-dev: 2,839 BPE tokens [chars/4: 3,135] (standard ~), 14 sections, 4 code blocks
[eval-driven-dev]    ⚠  Skill is 2,839 BPE tokens (chars/4 estimate: 3,135) — approaching "comprehensive" range where gains diminish.
[eval-driven-dev]    ⚠  No numbered workflow steps — agents follow sequenced procedures more reliably.
�[32m✅ All checks passed (1 skill(s))�[0m

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the eval-driven-dev skill to a newer pixie-qa version and restructures the workflow into smaller, step-scoped reference documents, with helper scripts for setup and background process management.

Changes:

  • Bumped skill metadata to version: 0.4.0 and documented pixie-qa version/source.
  • Refactored the skill workflow into Steps 1–7 with per-step reference docs (and added generated API references).
  • Added resources/ shell scripts to automate setup, background server start, and server stop; updated docs/README.skills.md references accordingly.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
skills/eval-driven-dev/SKILL.md Updates workflow structure, adds setup/stop guidance, bumps metadata/version.
skills/eval-driven-dev/resources/setup.sh Automates skill update, pixie-qa install/upgrade, pixie init, and starts web UI server.
skills/eval-driven-dev/resources/stop-server.sh Attempts to stop pixie web UI server based on a lock-file/port.
skills/eval-driven-dev/resources/run-with-timeout.sh Utility to run a command in background and kill it after a timeout.
skills/eval-driven-dev/references/understanding-app.md Removes legacy Step 1 reference (replaced by new Step 1a/1b/1c refs).
skills/eval-driven-dev/references/run-harness-patterns.md Removes legacy run harness patterns reference (replaced by new Step 3 + examples).
skills/eval-driven-dev/references/pixie-api.md Removes legacy pixie API reference (replaced by generated API refs).
skills/eval-driven-dev/references/instrumentation.md Removes legacy instrumentation reference (replaced by Step 2 + instrumentation API ref).
skills/eval-driven-dev/references/eval-tests.md Removes legacy eval tests reference (replaced by Step 4–6 + generated refs).
skills/eval-driven-dev/references/dataset-generation.md Removes legacy dataset generation reference (replaced by Step 5 ref).
skills/eval-driven-dev/references/1-a-entry-point.md Adds new Step 1a entry-point artifact guidance.
skills/eval-driven-dev/references/1-b-data-flow.md Adds new Step 1b DAG artifact guidance.
skills/eval-driven-dev/references/1-c-eval-criteria.md Adds new Step 1c eval criteria artifact guidance.
skills/eval-driven-dev/references/2-instrument-and-observe.md Adds new Step 2 instrumentation/trace/DAG validation workflow.
skills/eval-driven-dev/references/3-run-harness.md Adds new Step 3 run harness contract + pointers to examples.
skills/eval-driven-dev/references/run-harness-examples/fastapi-web-server.md Adds FastAPI harness example and concurrency/lifespan notes.
skills/eval-driven-dev/references/run-harness-examples/cli-app.md Adds CLI harness example.
skills/eval-driven-dev/references/run-harness-examples/standalone-function.md Adds standalone-function harness example.
skills/eval-driven-dev/references/4-define-evaluators.md Adds new Step 4 evaluator mapping + custom evaluator guidance.
skills/eval-driven-dev/references/evaluators.md Adds generated built-in evaluator catalog and rules.
skills/eval-driven-dev/references/testing-api.md Adds generated testing API reference and dataset JSON format.
skills/eval-driven-dev/references/instrumentation-api.md Adds generated instrumentation API reference and CLI commands.
skills/eval-driven-dev/references/5-build-dataset.md Adds new Step 5 dataset JSON build/validate guidance.
skills/eval-driven-dev/references/6-run-tests.md Adds new Step 6 test execution + analysis guidance.
skills/eval-driven-dev/references/7-investigation.md Updates investigation workflow to Step 7 and adds a stop/continue gate.
docs/README.skills.md Updates listed reference files/dirs for the skill.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants