feat(tooling): Add YAML-driven qualification harness.#168
Open
Charly-sketch wants to merge 5 commits into
Open
feat(tooling): Add YAML-driven qualification harness.#168Charly-sketch wants to merge 5 commits into
Charly-sketch wants to merge 5 commits into
Conversation
Adds a human-in-the-loop qualification test tier driven by YAML scenarios. The runner: - flashes a per-driver qualification sketch - opens serial before reset (OpenOCD) - walks interactive phases declared in YAML - captures measurements per phase - evaluates assertions (min, max, delta_vs, delta_min, delta_max) - archives full session logs Make integration: - auto-generated `qualify-<driver>` targets - `make list-qualification` for discovery Includes a reference scenario for HTS221. This introduces a fourth validation layer beyond unit/integration tests: ensuring that sensors react coherently to real-world perturbations. Closes #124
There was a problem hiding this comment.
Pull request overview
This PR adds a new “qualification” validation tier to the repo’s tooling: YAML-driven, human-in-the-loop sessions that flash a qualification sketch, capture serial measurements during operator-guided phases, evaluate declarative assertions, and archive a full session log under logs/qualification/.
Changes:
- Introduces
scripts/qualify.pyto run interactive, YAML-defined qualification scenarios (capture + assertion evaluation + log archival). - Adds Makefile integration to discover scenarios under
tests/qualification/*and exposelist-qualification/qualify-<driver>targets. - Adds initial HTS221 reference scenario (
qualify.yaml+qualify.ino) and updates testing documentation; adds PyYAML to Python constraints.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
tests/TESTING.md |
Documents the new qualification tier and how to run scenarios. |
tests/qualification/hts221/qualify.yaml |
Defines an example HTS221 interactive qualification scenario with delta/min/max assertions. |
tests/qualification/hts221/qualify.ino |
Streams machine-readable HTS221 measurements for the harness to capture. |
scripts/qualify.py |
Implements the YAML-driven qualification runner (flash/reset/capture/assert/log). |
requirements.txt |
Pins PyYAML for the new runner. |
Makefile |
Auto-discovers qualification scenarios and generates qualify-<driver> targets. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Wrap phase names in brackets in all log records to avoid ambiguity when phase names contain spaces.
Run OpenOCD with cwd=ROOT so relative config paths (interface/..., target/...) resolve correctly regardless of invocation directory. Use absolute path to .venv/bin/pio and run with cwd=ROOT so the qualification runner works regardless of invocation location. Emit clear error when <signal>_delta_vs is missing or invalid instead of reporting a confusing 'None' reference phase.
Make qualify-<driver> depend on setup so the virtualenv and Python dependencies (PyYAML, pyserial) are always available on fresh checkouts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #124
Closes #125
Summary
Add a fourth validation tier to the tooling: YAML-driven human-in-the-loop qualification sessions for checking that a real sensor reacts coherently to real-world perturbations.
Unlike unit tests and scripted hardware captures, qualification tests intentionally involve an operator (breathe on the sensor, warm the board, cover it, etc.) and compare the observed measurements against declarative per-phase assertions.
This PR introduces the generic Python harness, Makefile integration, session logging, and a first reference scenario for HTS221.
Changes
Added
scripts/qualify.py, a YAML-driven qualification runner that:tests/qualification/<driver>/qualify.yamlqualify.inomin,max,delta_vs,delta_min,delta_max)logs/qualification/Added
PyYAMLtorequirements.txtAdded dynamic Makefile qualification integration:
make list-qualificationmake qualify-<driver>targets fromtests/qualification/*Added qualification documentation section to
tests/TESTING.mdAdded first end-to-end reference qualification scenario for HTS221:
tests/qualification/hts221/qualify.yamltests/qualification/hts221/qualify.inoAdded
logs/qualification/.gitkeepfor archived session outputsReused the existing flash/reset plumbing introduced by the recent tooling stack (
flash-*/capture-*family)Checklist
make lintpasses (clang-format)make buildpasses (PlatformIO)make test-nativepasses (if native tests exist)make test-hardwarepasses on a connected STeaMi (if hardware tests exist)