Skip to content

Commit bbfe36c

Browse files
HelloThisWorldHelloThisWorld
authored
CI: verify agent skills against each build with Agent Skill Verifier (#11)
* ci: verify agent skills against each build with Agent Skill Verifier Adds a skill-verification job that, after building the CLI: 1. rebuilds the skill fixture workspace from this build's CLI output and diffs it against the grounding facts committed in agent-skill-verification-template@v0.1.0 (snapshots, WORKSPACE.md, specs, manifests) — CLI output drift that would break the specbridge-* skills' citations now fails CI with a remediation hint; 2. statically validates all 11 specbridge-* skill contracts and their evaluation cases with a pinned, checksum-verified agent-skill-verifier v0.1.0 release binary. Deterministic and offline apart from two pinned GitHub downloads; full model-driven eval runs remain a local/manual step. Also sets explicit read-only workflow permissions. * ci: compare runner-list snapshot on environment-independent fields only The committed skill-verification snapshot embeds machine-dependent capability probes (runner CLIs installed, versions, authentication) from the workstation that generated it; CI runners have none of those installed, so byte-equality failed. runner-list is now excluded from the byte-level drift diff and checked by a dedicated script that compares the CLI-determined fields (profile set, implementation, enablement) and ignores the environment probes. A genuine CLI regression in the runner catalog still fails; a machine difference no longer does. * ci: standalone skill-test workflow with README status badge Moves the skill-verification job into its own workflow (name: 'skill test') so the GitHub status badge reflects exactly the skill-test outcome, and adds that badge to the README linking to the agent-skill-verification-template repository. Job content is unchanged.
1 parent b062d4f commit bbfe36c

4 files changed

Lines changed: 162 additions & 0 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,9 @@ on:
55
branches: [main, master]
66
pull_request:
77

8+
permissions:
9+
contents: read
10+
811
# Everything below runs fully offline after dependency install:
912
# no LLM, no API key, no external service.
1013
jobs:
@@ -64,3 +67,4 @@ jobs:
6467
- name: Claude Code plugin bundle is reproducible
6568
if: matrix.os == 'ubuntu-latest'
6669
run: git diff --exit-code integrations/claude-code-plugin/specbridge/dist
70+

‎.github/workflows/skill-verification.yml‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
# Verifies the 11 specbridge-* agent skills against each build of the CLI
2+
# using the Agent Skill Verifier
3+
# (https://github.com/HelloThisWorld/agent-skill-verification-template):
4+
#
5+
# 1. the skill fixture workspace is rebuilt from this build's CLI output
6+
# and diffed against the committed grounding facts (snapshots,
7+
# WORKSPACE.md, spec files) — a CLI output change that would break the
8+
# skills' citations fails here;
9+
# 2. every skill contract + evaluation case file is statically validated
10+
# with a pinned, checksum-verified agent-skill-verifier release.
11+
#
12+
# Full model-driven eval runs (verify --adapter llm) stay a local/manual
13+
# step because they need a live model endpoint; everything here is
14+
# deterministic and offline apart from the two pinned GitHub downloads.
15+
#
16+
# This is a standalone workflow (not a job in ci.yml) so the README badge
17+
# reflects exactly the skill-test outcome.
18+
name: skill test
19+
20+
on:
21+
push:
22+
branches: [main, master]
23+
pull_request:
24+
25+
permissions:
26+
contents: read
27+
28+
jobs:
29+
skill-verification:
30+
name: agent skill verification
31+
runs-on: ubuntu-latest
32+
env:
33+
# Pinned verifier release. Update the version and the asset SHA-256
34+
# (from the release's SHA256SUMS.txt) together.
35+
ASV_VERSION: 0.1.0
36+
ASV_SHA256: 2307aa1ee6a8698fe5d7dfc12ef9413ce086e5f3fe3748501abd0d10cfe68084
37+
# Ref of the verification template holding the specbridge skill
38+
# contracts, evaluation cases, and fixture builder.
39+
TEMPLATE_REF: v0.1.0
40+
steps:
41+
- uses: actions/checkout@v4
42+
43+
- uses: pnpm/action-setup@v4
44+
# Version comes from the "packageManager" field in package.json.
45+
46+
- uses: actions/setup-node@v4
47+
with:
48+
node-version: 22
49+
cache: pnpm
50+
51+
- name: Install dependencies
52+
run: pnpm install --frozen-lockfile
53+
54+
- name: Build the SpecBridge CLI
55+
run: pnpm build
56+
57+
- name: Checkout the verification template (pinned ref)
58+
uses: actions/checkout@v4
59+
with:
60+
repository: HelloThisWorld/agent-skill-verification-template
61+
ref: ${{ env.TEMPLATE_REF }}
62+
path: verification-template
63+
64+
- name: Download the pinned agent-skill-verifier release and verify its checksum
65+
run: |
66+
curl -fsSLO "https://github.com/HelloThisWorld/agent-skill-verification-template/releases/download/v${ASV_VERSION}/agent-skill-verifier-v${ASV_VERSION}-linux-x64.tar.gz"
67+
echo "${ASV_SHA256} agent-skill-verifier-v${ASV_VERSION}-linux-x64.tar.gz" | sha256sum -c -
68+
mkdir -p .asv
69+
tar -xzf "agent-skill-verifier-v${ASV_VERSION}-linux-x64.tar.gz" -C .asv
70+
.asv/agent-skill-verifier --version
71+
72+
- name: Rebuild the skill fixture from this build's CLI (grounding drift check)
73+
env:
74+
SPECBRIDGE_REPO: ${{ github.workspace }}
75+
run: |
76+
node verification-template/scripts/build-specbridge-fixture.mjs
77+
# runner-list contains machine-dependent capability probes, so it is
78+
# compared on its environment-independent fields by the dedicated
79+
# script instead of byte equality.
80+
git -C verification-template show HEAD:fixtures/specbridge-workspace/snapshots/runner-list.json > /tmp/runner-list-committed.json
81+
node scripts/check-skill-runner-snapshot.mjs \
82+
/tmp/runner-list-committed.json \
83+
verification-template/fixtures/specbridge-workspace/snapshots/runner-list.json
84+
# Timestamps, install-record ids, and local archive paths differ per
85+
# rebuild by design; every other grounded fact (snapshots,
86+
# WORKSPACE.md, specs, manifests, hashes) must be byte-identical.
87+
if ! git -C verification-template diff --exit-code -- \
88+
fixtures/specbridge-workspace \
89+
':(exclude)fixtures/specbridge-workspace/.specbridge/extensions/grants.json' \
90+
':(exclude)fixtures/specbridge-workspace/.specbridge/extensions/records.jsonl' \
91+
':(exclude)fixtures/specbridge-workspace/.specbridge/extensions/state.json' \
92+
':(exclude)fixtures/specbridge-workspace/snapshots/runner-list.json'; then
93+
echo "::error::SpecBridge CLI output drifted from the committed skill-verification fixture. If the change is intentional, regenerate the fixture in agent-skill-verification-template (scripts/build-specbridge-fixture.mjs) and pin a new TEMPLATE_REF."
94+
exit 1
95+
fi
96+
97+
- name: Validate every specbridge skill with agent-skill-verifier
98+
working-directory: verification-template
99+
run: |
100+
fail=0
101+
for dir in skills/specbridge-*/; do
102+
name="$(basename "$dir")"
103+
for cases in "testcases/${name}.json" "testcases/${name}-negative.json"; do
104+
[ -f "$cases" ] || continue
105+
if "$GITHUB_WORKSPACE/.asv/agent-skill-verifier" validate --skill "$dir" --cases "$cases" --quiet; then
106+
echo "ok: ${name} / $(basename "$cases")"
107+
else
108+
echo "::error::skill validation failed: ${name} / $(basename "$cases")"
109+
fail=1
110+
fi
111+
done
112+
done
113+
exit "$fail"

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# SpecBridge
22

33
[![CI](https://github.com/HelloThisWorld/specbridge/actions/workflows/ci.yml/badge.svg)](https://github.com/HelloThisWorld/specbridge/actions/workflows/ci.yml)
4+
[![skill test](https://github.com/HelloThisWorld/specbridge/actions/workflows/skill-verification.yml/badge.svg)](https://github.com/HelloThisWorld/agent-skill-verification-template)
45
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
56

67
An open, model-agnostic spec runtime for existing Kiro projects.

‎scripts/check-skill-runner-snapshot.mjs‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
// Compare the environment-independent fields of a SpecBridge `runner list`
2+
// snapshot against a freshly generated one: the profile set, each profile's
3+
// implementation, and its enablement. Capability probes (capabilities,
4+
// version, authentication) legitimately differ per machine — the committed
5+
// skill-verification snapshot was generated on a workstation with runner
6+
// CLIs installed, while CI runners have none — so those fields are ignored.
7+
//
8+
// Used by the skill-verification CI job together with the byte-level drift
9+
// check that covers every other fixture file.
10+
//
11+
// usage: node scripts/check-skill-runner-snapshot.mjs <committed.json> <rebuilt.json>
12+
import { readFileSync } from "node:fs";
13+
14+
const [committedPath, rebuiltPath] = process.argv.slice(2);
15+
if (!committedPath || !rebuiltPath) {
16+
console.error("usage: node scripts/check-skill-runner-snapshot.mjs <committed.json> <rebuilt.json>");
17+
process.exit(2);
18+
}
19+
20+
const normalize = (path) => {
21+
const raw = JSON.parse(readFileSync(path, "utf8"));
22+
return (raw.profiles ?? [])
23+
.map((profile) => ({
24+
profile: profile.profile,
25+
implementation: profile.implementation,
26+
enabled: profile.enabled,
27+
}))
28+
.sort((a, b) => a.profile.localeCompare(b.profile));
29+
};
30+
31+
const committed = JSON.stringify(normalize(committedPath), null, 2);
32+
const rebuilt = JSON.stringify(normalize(rebuiltPath), null, 2);
33+
34+
if (committed !== rebuilt) {
35+
console.error("runner-list drift in environment-independent fields (profile/implementation/enabled):");
36+
console.error("--- committed (verification template)");
37+
console.error(committed);
38+
console.error("+++ rebuilt (this build)");
39+
console.error(rebuilt);
40+
process.exit(1);
41+
}
42+
console.log(
43+
`runner-list: ${normalize(rebuiltPath).length} profiles match the committed snapshot on all environment-independent fields.`,
44+
);

0 commit comments

Comments
 (0)