test(skills): add A/B usability harness comparing the same prompt with and without skills, plus static skill validation

Author: francescobianco

.gitignore

@@ -28,3 +28,4 @@ Homestead.json
 .env.production
 .phpactor.json
 auth.json
+test/results/

test/README.md

@@ -0,0 +1,37 @@
+# Test environment
+
+A minimal environment proving that the **skills are usable by an agent** — not that the Openapi APIs work. The core test is an A/B comparison: the same prompt run by an agent *without* the skills and by an agent *with* them, so the difference the skills make is directly observable.
+
+## 1. A/B usability comparison — `run_comparison.sh`
+
+**Problem given to both agents** ([prompt.txt](prompt.txt)): "All we know about a supplier is its VAT number 12485671007 — find the company name, status, registered office, and the province/region of its ZIP, using the Openapi platform with the credentials in the environment."
+
+The harness:
+
+1. creates two throwaway workspaces under `test/results/<timestamp>/`:
+   - **bare** — empty folder, no skills
+   - **skilled** — identical, but with `skills/` installed as project skills (`.claude/skills/`)
+2. runs `claude -p` headless with the same prompt and the same allowed tools in both
+3. saves both transcripts and prints a comparison table: wall time, turns, cost, and whether a `SOLUTION` block was produced
+
+```bash
+export OPENAPI_EMAIL=you@example.com
+export OPENAPI_KEY=...
+bash test/run_comparison.sh
+```
+
+What to look for in the transcripts: the *bare* agent has to discover authentication (Basic vs Bearer, the scope format `METHOD:host/path`), guess endpoint names, and typically burns turns on 401/404s. The *skilled* agent reads `openapi-auth`, `openapi-company`, `openapi-geo` and goes straight to: scoped token → `IT-start/{vat}` → `cap/{zip}` → solution.
+
+Notes: both agents make real (cheap, but paid) API calls; results land in `test/results/` which is git-ignored.
+
+## 2. Static validation — `validate_skills.sh`
+
+Sanity lint for every `skills/*/SKILL.md` — no credentials needed, exits non-zero on violations:
+
+- frontmatter present with `name` (matching the folder) and `description` (≤ 1024 chars)
+- no references to `knowledge/` (skills must be self-contained — see repo README)
+- mentions `openapi-auth` for authentication
+
+```bash
+bash test/validate_skills.sh
+```

test/prompt.txt

@@ -0,0 +1,14 @@
+All we know about a supplier is its Italian VAT number: 12485671007.
+
+Using the Openapi (openapi.com) platform APIs, find out:
+1. the company name and whether it is still active
+2. the address of its registered office
+3. the province and region its ZIP code belongs to
+
+Account credentials are available in the environment variables OPENAPI_EMAIL and OPENAPI_KEY. Work only from the command line (curl/jq). Be frugal: paid requests cost wallet credit, so make the minimum number of API calls needed.
+
+End your answer with a block:
+SOLUTION:
+  company: <name> (<status>)
+  office: <address>
+  territory: <province>, <region>

test/run_comparison.sh

@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+# A/B usability test for the skills.
+#
+# The same prompt (prompt.txt) is given to two headless Claude Code agents:
+#   A. "bare"    — empty workspace, NO skills installed
+#   B. "skilled" — identical workspace WITH skills/ installed as project skills
+#
+# What is being tested is NOT the Openapi APIs, but whether the skills let an
+# agent attack the problem and drive it straight to the solution: compare the
+# two transcripts (number of calls, wrong turns, time, correctness).
+#
+# Requirements: claude CLI, OPENAPI_EMAIL + OPENAPI_KEY in the environment.
+set -u
+
+TEST_DIR="$(cd "$(dirname "$0")" && pwd)"
+REPO_DIR="$(dirname "$TEST_DIR")"
+RUN_DIR="$TEST_DIR/results/$(date +%Y%m%d-%H%M%S)"
+PROMPT="$(cat "$TEST_DIR/prompt.txt")"
+
+command -v claude >/dev/null || { echo "ERROR: claude CLI not found"; exit 1; }
+[ -n "${OPENAPI_EMAIL:-}" ] && [ -n "${OPENAPI_KEY:-${OPENAPI_API_KEY:-}}" ] \
+    || { echo "ERROR: OPENAPI_EMAIL and OPENAPI_KEY must be set"; exit 1; }
+
+mkdir -p "$RUN_DIR/bare" "$RUN_DIR/skilled/.claude"
+cp -r "$REPO_DIR/skills" "$RUN_DIR/skilled/.claude/skills"
+
+run_agent() { # $1 = variant dir name
+    local variant="$1"
+    echo "=== Running '$variant' agent (this may take a few minutes)..."
+    ( cd "$RUN_DIR/$variant" && \
+      /usr/bin/time -o ../"$variant".time -f '%e' \
+      claude -p "$PROMPT" \
+          --allowedTools "Bash,Read,Glob,Grep" \
+          --output-format json \
+          > ../"$variant".json 2> ../"$variant".err )
+    jq -r '.result // "NO RESULT"' "$RUN_DIR/$variant.json" > "$RUN_DIR/$variant.answer.md"
+}
+
+run_agent bare
+run_agent skilled
+
+# ------------------------------------------------------------------ report
+report() { # $1 = variant
+    local v="$1" j="$RUN_DIR/$1.json"
+    printf '%-10s | %8ss | %5s turns | $%-7s | solution: %s\n' \
+        "$v" \
+        "$(cat "$RUN_DIR/$v.time" 2>/dev/null || echo '?')" \
+        "$(jq -r '.num_turns // "?"' "$j")" \
+        "$(jq -r '.total_cost_usd // "?" | if type=="number" then (.*10000|round)/10000 else . end' "$j")" \
+        "$(grep -q 'SOLUTION' "$RUN_DIR/$v.answer.md" && echo yes || echo NO)"
+}
+
+echo
+echo "==================== COMPARISON ===================="
+report bare
+report skilled
+echo "====================================================="
+echo
+echo "Full answers:"
+echo "  $RUN_DIR/bare.answer.md"
+echo "  $RUN_DIR/skilled.answer.md"
+echo
+echo "Compare them side by side, e.g.:"
+echo "  diff -y --width 160 $RUN_DIR/bare.answer.md $RUN_DIR/skilled.answer.md"

test/validate_skills.sh

@@ -0,0 +1,42 @@
+#!/usr/bin/env bash
+# Static validation: every skill is well-formed and respects the
+# knowledge/skills separation rule documented in the repo README.
+set -u
+
+REPO_DIR="$(cd "$(dirname "$0")/.." && pwd)"
+SKILLS_DIR="$REPO_DIR/skills"
+errors=0
+
+fail() { echo "  FAIL: $1"; errors=$((errors + 1)); }
+
+for skill_md in "$SKILLS_DIR"/*/SKILL.md; do
+    dir_name="$(basename "$(dirname "$skill_md")")"
+    echo "Checking $dir_name"
+
+    head -1 "$skill_md" | grep -q '^---$' || fail "file must start with '---' (YAML frontmatter)"
+
+    frontmatter="$(awk '/^---$/{n++; next} n==1{print} n>1{exit}' "$skill_md")"
+
+    name="$(echo "$frontmatter" | sed -n 's/^name:[[:space:]]*//p')"
+    [ -n "$name" ] || fail "frontmatter is missing 'name'"
+    [ "$name" = "$dir_name" ] || fail "frontmatter name '$name' does not match folder '$dir_name'"
+
+    description="$(echo "$frontmatter" | sed -n 's/^description:[[:space:]]*//p')"
+    [ -n "$description" ] || fail "frontmatter is missing 'description'"
+    [ "${#description}" -le 1024 ] || fail "description exceeds 1024 characters"
+
+    if grep -qn 'knowledge/' "$skill_md"; then
+        fail "skill references knowledge/ — skills must be self-contained (see README)"
+    fi
+
+    grep -q 'openapi-auth' "$skill_md" || [ "$dir_name" = "openapi-auth" ] \
+        || fail "skill does not mention the openapi-auth skill for authentication"
+done
+
+echo
+if [ "$errors" -eq 0 ]; then
+    echo "OK: all skills are valid."
+else
+    echo "$errors problem(s) found."
+    exit 1
+fi