|
| 1 | +--- |
| 2 | +name: provider-compat |
| 3 | +description: Audit MessagePit provider stub fidelity against the real provider API spec. Infers claimed scope from Go source, fetches the real spec, generates compat_test.go, runs tests, and reports a fidelity score. |
| 4 | +triggers: |
| 5 | + - /provider-compat |
| 6 | +allowed-tools: Bash, Read, Write, Edit, Agent, WebFetch, WebSearch |
| 7 | +--- |
| 8 | + |
| 9 | +# /provider-compat |
| 10 | + |
| 11 | +Audit how closely the MessagePit stub for `$ARGUMENTS` matches the real provider API. |
| 12 | + |
| 13 | +## Script |
| 14 | + |
| 15 | +```bash |
| 16 | +SKILL_DIR=$(find . -path "*/.claude/skills/provider-compat/scripts/provider_compat.py" | head -1) |
| 17 | +``` |
| 18 | + |
| 19 | +All heavy lifting goes through the script. Run from the project root. |
| 20 | + |
| 21 | +## Workflow |
| 22 | + |
| 23 | +### 1. Resolve Provider |
| 24 | + |
| 25 | +```bash |
| 26 | +python3 "$SKILL_DIR" --providers |
| 27 | +``` |
| 28 | + |
| 29 | +Match `$ARGUMENTS` (lowercase, trimmed) against the registry. If not found, print the registry and exit. |
| 30 | + |
| 31 | +### 2. Infer Claimed Scope |
| 32 | + |
| 33 | +```bash |
| 34 | +python3 "$SKILL_DIR" --scope <name> |
| 35 | +``` |
| 36 | + |
| 37 | +This reads all non-test `.go` files in the provider package and returns JSON: |
| 38 | + |
| 39 | +```json |
| 40 | +{ |
| 41 | + "provider": "sendgrid", |
| 42 | + "package": "internal/sendgrid", |
| 43 | + "port": 8101, |
| 44 | + "spec_url": "https://...", |
| 45 | + "scope": { |
| 46 | + "routes": [{"path": "/v3/mail/send", "methods": ["POST"]}], |
| 47 | + "auth": "Bearer token", |
| 48 | + "request_fields": ["content", "from", "personalizations", "subject", ...], |
| 49 | + "status_codes": [202, 400, 401, 413, 500] |
| 50 | + } |
| 51 | +} |
| 52 | +``` |
| 53 | + |
| 54 | +Read this output — it is the **Claimed Scope** for the rest of the workflow. |
| 55 | + |
| 56 | +### 3. Fetch the Real Spec |
| 57 | + |
| 58 | +Delegate to the `spec-analyst` subagent (`.claude/agents/spec-analyst.md`), passing: |
| 59 | +- The `spec_url` from step 2 |
| 60 | +- The full claimed scope JSON |
| 61 | + |
| 62 | +The subagent returns a **Spec Snapshot** — the real API's contract for the claimed endpoints, distilled to the fields and status codes that matter. |
| 63 | + |
| 64 | +If the `spec-analyst` agent is unavailable, use `WebFetch` inline and extract the relevant paths yourself. |
| 65 | + |
| 66 | +### 4. Generate `compat_test.go` |
| 67 | + |
| 68 | +Get the file header: |
| 69 | + |
| 70 | +```bash |
| 71 | +python3 "$SKILL_DIR" --write-header <name> |
| 72 | +``` |
| 73 | + |
| 74 | +Write `internal/<name>/compat_test.go` — **fully regenerated on every run**. |
| 75 | + |
| 76 | +Structure: |
| 77 | + |
| 78 | +```go |
| 79 | +// Code generated by /provider-compat. DO NOT EDIT. |
| 80 | +// Re-run: /provider-compat <name> |
| 81 | + |
| 82 | +package <name>_test |
| 83 | + |
| 84 | +import ( |
| 85 | + "net/http" |
| 86 | + "net/http/httptest" |
| 87 | + "strings" |
| 88 | + "testing" |
| 89 | + |
| 90 | + "github.com/coreydaley/messagepit/internal/<name>" |
| 91 | + "github.com/coreydaley/messagepit/internal/storage" |
| 92 | + "github.com/coreydaley/messagepit/logger" |
| 93 | +) |
| 94 | + |
| 95 | +func init() { |
| 96 | + logger.NoLogging = true |
| 97 | +} |
| 98 | + |
| 99 | +func setupCompat(t *testing.T) { |
| 100 | + t.Helper() |
| 101 | + storage.InitDB() |
| 102 | + t.Cleanup(storage.Close) |
| 103 | +} |
| 104 | +``` |
| 105 | + |
| 106 | +Then one `TestCompat_*` function per claimed endpoint. Each test: |
| 107 | + |
| 108 | +1. Calls `setupCompat(t)` |
| 109 | +2. Builds the exact request the real spec requires (correct method, path, auth header, content-type, body with required fields) |
| 110 | +3. Posts it to `httptest.NewRecorder()` + the handler directly |
| 111 | +4. Asserts the success status code matches the spec |
| 112 | +5. Has a sub-test (`t.Run`) for each expected error case (missing auth → 401, malformed body → 400, etc.) |
| 113 | + |
| 114 | +Name tests `TestCompat_<METHOD>_<PathSuffix>` (e.g., `TestCompat_POST_MailSend`). |
| 115 | + |
| 116 | +Do **not** duplicate assertions already in `<name>_test.go`. Compat tests cover spec conformance; behavioral edge cases stay human-owned. |
| 117 | + |
| 118 | +### 5. Run the Tests |
| 119 | + |
| 120 | +```bash |
| 121 | +go test ./internal/<name>/... -run TestCompat_ -v 2>&1 |
| 122 | +``` |
| 123 | + |
| 124 | +Capture output. Parse pass/fail per test. |
| 125 | + |
| 126 | +### 6. Report |
| 127 | + |
| 128 | +Print a fidelity report: |
| 129 | + |
| 130 | +``` |
| 131 | +=== Provider Compat Report: <name> === |
| 132 | +
|
| 133 | +Claimed scope |
| 134 | + Endpoints : GET /path, POST /path ... |
| 135 | + Auth : Bearer token |
| 136 | + Fields : N request fields |
| 137 | + Statuses : 200, 400, 401 ... |
| 138 | +
|
| 139 | +Spec coverage |
| 140 | + Matched : N/M claimed endpoints found in spec |
| 141 | + Gaps : <claimed items not in spec — possible drift or extension> |
| 142 | +
|
| 143 | +Test results |
| 144 | + Passed : N |
| 145 | + Failed : N |
| 146 | + <for each failure: test name, assertion that failed, spec citation> |
| 147 | +
|
| 148 | +Not yet claimed (in spec, not in twin) |
| 149 | + <paths in scope domain not yet implemented> |
| 150 | +
|
| 151 | +Fidelity score: <passed>/<total> (<percent>%) |
| 152 | +``` |
| 153 | + |
| 154 | +End with `Twin is in sync.` if all pass, or `Action required: see failures above.` if any fail. |
0 commit comments