diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 120000 index 000000000..be77ac83a --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1 @@ +../AGENTS.md \ No newline at end of file diff --git a/.github/instructions/agent-files.instructions.md b/.github/instructions/agent-files.instructions.md new file mode 100644 index 000000000..0481329f1 --- /dev/null +++ b/.github/instructions/agent-files.instructions.md @@ -0,0 +1,30 @@ +--- +applyTo: ".github/skills/**/*.md,.github/instructions/**/*.md,AGENTS.md" +--- +# Agent Files — Caveman Style + +Files in `.github/skills/`, `.github/instructions/`, and `AGENTS.md` must use [caveman skill](./../skills/caveman/SKILL.md) to keep context short and tokens efficient. + +## Applies To + +- `AGENTS.md` +- `.github/skills/**/*.md` (skill files and support docs) +- `.github/instructions/**/*.md` (instruction files) + +## Caveman Rule + +All text except code blocks and security warnings: +- Drop articles (a/an/the) +- No filler (just, really, basically, actually, simply) +- No pleasantries (sure, certainly, of course, happy to) +- Fragments OK. Short synonyms (big not extensive) +- Technical terms exact +- Pattern: `[thing] [action] [reason]. [next step].` + +**Default intensity:** full + +See [caveman skill](./../skills/caveman/SKILL.md) for lite/ultra and examples. + +## Exception + +Security warnings + irreversible action confirmations → normal English. Resume caveman after warning complete. diff --git a/.github/instructions/cypher-queries.instructions.md b/.github/instructions/cypher-queries.instructions.md new file mode 100644 index 000000000..743e820da --- /dev/null +++ b/.github/instructions/cypher-queries.instructions.md @@ -0,0 +1,51 @@ +--- +applyTo: "**/*.cypher" +--- +# Cypher Query Conventions + +## Style + +- **First line: description comment** — format: `// Description sentence. Requires "Other_Query.cypher".` + Used in generated [CYPHER.md](../../CYPHER.md) reference +- Blank line between comment and statement +- One statement per file — no combining without explicit approval +- Standard Cypher where possible. Use APOC sparingly — confirm with user first +- Parameterize with `$parameterName` +- Keywords uppercase: `MATCH`, `WHERE`, `RETURN`, `WITH`, `SET`, `ORDER BY`, `LIMIT`, etc. +- Literals lowercase: `null`, `true`, `false` +- Each clause own line — no inline chaining +- `ORDER BY` and `LIMIT` on own lines +- Single quotes for strings: `'value'` not `"value"` +- No semicolons at end + +## Naming + +- Files: `Snake_Case_Description.cypher` +- Node labels: `PascalCase` +- Relationship types: `UPPER_SNAKE_CASE` +- Properties: `camelCase` +- No backtick-quoted identifiers — use clean label/property names + +## Spacing + +- One space between label predicate and property predicate: `(n:Label {prop: 1})` +- Space after each comma: `a, b, c` not `a,b,c` +- Wrapping space around operators: `a <> b`, `p = (s)-->(e)` +- No padding space inside function calls: `count(n)` not `count( n )` + +## Patterns + +- Anonymous nodes/rels when variable unused: `[:LIKES]` not `[r:LIKES]` + +## Verification + +Before finalizing query, verify against live Neo4j to catch syntax errors and warnings. +Requires Neo4j running — use [neo4j-management skill](./../skills/neo4j-management/SKILL.md) first. + +```shell +# From repo root — NEO4J_INITIAL_PASSWORD must be exported +./scripts/executeQuery.sh +``` + +- Empty result ≠ broken query — data may be absent +- `executeQueryFunctions.sh` for script composition only — not for ad-hoc verification diff --git a/.github/instructions/generated-reference-docs.instructions.md b/.github/instructions/generated-reference-docs.instructions.md new file mode 100644 index 000000000..48e199c86 --- /dev/null +++ b/.github/instructions/generated-reference-docs.instructions.md @@ -0,0 +1,28 @@ +--- +applyTo: "CYPHER.md,SCRIPTS.md,ENVIRONMENT_VARIABLES.md" +--- +# Generated Reference Files + +[CYPHER.md](../../CYPHER.md), [SCRIPTS.md](../../SCRIPTS.md), [ENVIRONMENT_VARIABLES.md](../../ENVIRONMENT_VARIABLES.md) — **auto-generated**. Do not edit directly. Changes get overwritten. + +## Do Not Run Generators Manually + +Generators run automatically in pipeline — output **auto-committed**. Running locally, including result in commit causes **git merge conflicts** in pipeline. Only run generator if user explicitly insists. + +## Influence Indirectly + +| File | Edit instead | +|------|--------------| +| `CYPHER.md` | First line `// Description.` of each `.cypher` file | +| `SCRIPTS.md` | First line `# Description.` of each `.sh` file | +| `ENVIRONMENT_VARIABLES.md` | Inline `# comment` after env var declaration in `.sh` files | + +Moved/renamed files detected and reflected automatically on next pipeline run. + +## Generators + +See [scripts/documentation/](../../scripts/documentation/): + +- `generateCypherReference.sh` → `CYPHER.md` +- `generateScriptReference.sh` → `SCRIPTS.md` +- `generateEnvironmentVariableReference.sh` + `appendEnvironmentVariables.sh` → `ENVIRONMENT_VARIABLES.md` diff --git a/.github/instructions/python-dependencies.instructions.md b/.github/instructions/python-dependencies.instructions.md new file mode 100644 index 000000000..0d4b59c58 --- /dev/null +++ b/.github/instructions/python-dependencies.instructions.md @@ -0,0 +1,49 @@ +--- +applyTo: "requirements.txt,conda-environment.yml" +--- +# Python Dependency Conventions + +Applies to [requirements.txt](../../requirements.txt) and [conda-environment.yml](../../conda-environment.yml). + +## Sync Rule + +Both files must stay **in sync**: same packages, same pinned versions. Package in one must be in other. + +Known exceptions (document inline with comment if they differ): +- `conda-environment.yml` may split packages (e.g. `nbconvert` + `nbconvert-webpdf` vs. `nbconvert[webpdf]` in pip) +- `conda-environment.yml` may use `pip:` block for packages not on conda-forge + +## Versioning + +- Pin all versions — no ranges, no `>=`, no unpinned entries +- `requirements.txt`: `==` (e.g. `pandas==2.2.3`) +- `conda-environment.yml`: `=` (e.g. `pandas=2.2.3`) + +## Adding or Updating + +- **Ask user before introducing new dependencies** +- Update both files when changing version + +## Verification + +**`requirements.txt`** — venv dry-run from [.github/workflows/internal-check-python-venv-support.yml](../../.github/workflows/internal-check-python-venv-support.yml): + +```shell +python -m venv .venv +source ./scripts/activatePythonEnvironment.sh +.venv/bin/pip install --dry-run --quiet --requirement requirements.txt +! .venv/bin/pip install --dry-run --requirement requirements.txt 2>/dev/null | grep -q "Would install" +``` + +**`conda-environment.yml`** — via [scripts/activateCondaEnvironment.sh](../../scripts/activateCondaEnvironment.sh): + +```shell +source ./scripts/activateCondaEnvironment.sh +``` + +Fallback (conda directly): + +```shell +conda env update --file conda-environment.yml --prune +conda activate codegraph +``` diff --git a/.github/instructions/python.instructions.md b/.github/instructions/python.instructions.md new file mode 100644 index 000000000..aa0dc1e21 --- /dev/null +++ b/.github/instructions/python.instructions.md @@ -0,0 +1,56 @@ +--- +applyTo: "**/*.py,**/*.ipynb" +--- +# Python Conventions + +Applies to Jupyter notebooks and standalone `.py` scripts. + +## Style + +- Type annotations everywhere — parameters, return types, non-obvious variables +- Verify with Pylance (strict mode preferred) +- No abbreviations. Write out names. `figure` not `fig`. `axis` not `ax`. `i`, `j`, `k` for indices OK +- Prefer named functions over code block comments. Comments explain *why*, not *what* +- Split into meaningful functions — ideally pure, easy to test +- Apply basic functional programming where it improves readability (map, filter, list comprehensions) +- f-strings preferred over `%` or `.format()` for string interpolation +- No `+` concatenation in loops — use `"".join()` or list accumulation +- `is`/`is not` for `None` comparisons, never `==`/`!=` +- Truthiness: `if seq:` not `if len(seq) > 0:`; `if x is not None:` not `if x != None:` +- Use `pathlib.Path` over `os.path` for file operations + +## Type Annotations + +- `X | None` over `Optional[X]`; `X | Y` over `Union[X, Y]` (PEP 604) +- Abstract container types in signatures: `Sequence` over `list`, `Mapping` over `dict` + +## Imports + +- Order: stdlib → third-party → local; one blank line between groups +- No wildcard imports (`from x import *`) +- One import statement per module line; multiple names from same module OK: `from x import A, B` + +## Error Handling + +- No bare `except:` — always name specific exceptions (`except ValueError:`) +- No mutable default arguments: `def f(x: list = [])` — use `None` + guard +- Use `with` for all file/resource management; never manual open/close + +## Naming + +- No generic names — no "util", "helper", "common", "shared" +- Module-level constants: `UPPER_CASE_WITH_UNDERSCORES` +- DRY only for domain-related concerns +- No magic numbers — extract named constants for literal comparisons + +## Docstrings + +- Public functions require `"""..."""` docstring (triple double quotes) +- First line imperative mood, ends with period: `"""Compute distances from center."""` +- Multi-line: blank line after summary, then `Args:`, `Returns:`, `Raises:` sections as needed + +## Notebook to Script Conversion + +- Extract charts and tables into standalone `.py` script with `main()` entry point +- Original notebook → `explore/` with metadata `"code_graph_analysis_pipeline_data_validation": "ValidateAlwaysFalse"` and "Exploration" added to title +- Charts → SVG, tables → CSV, descriptions → Markdown summary report diff --git a/.github/instructions/shell-scripts.instructions.md b/.github/instructions/shell-scripts.instructions.md new file mode 100644 index 000000000..b88e145c7 --- /dev/null +++ b/.github/instructions/shell-scripts.instructions.md @@ -0,0 +1,78 @@ +--- +applyTo: "**/*.sh" +--- +# Shell Script Conventions + +Target: **bash**. Must work on macOS, Linux, Windows (WSL/Git Bash). + +## Header + +- **First line: description comment** — used by [SCRIPTS.md](../../SCRIPTS.md). Format: `# Description sentence. Requires "other_script.sh".` +- Include required env vars, required tools, usage examples in description or `usage()` function +- Blank line after initial comment before code starts + +## Shebang and Strict Mode + +- First executable line: `#!/usr/bin/env bash` +- Enable strict mode immediately after: `set -euo pipefail` +- Set safe word-splitting: `IFS=$'\n\t'` +- `pipefail` propagates pipe failures — `set -e` bypassed in subshells and command groups; document where this applies + +## Style + +- POSIX-compliant as reasonable +- Always `"${variable}"` — curly braces + double quotes on every expansion +- Use `$(...)` for command substitution — not backticks +- No abbreviations. Write out variable and function names. `i`, `j`, `k` for indices OK +- Prefer named functions over code block comments. Comments explain *why*, not *what* +- Split into meaningful functions — ideally pure, easy to reason about +- Apply basic functional programming where it improves readability (piping, composition) + +## Naming + +- No generic names — no "util", "helper", "common", "shared" +- DRY only for domain-related concerns. Don't extract one-off operations + +## Variables and Scope + +- `local` for function-scoped variables +- `readonly` for constants +- `${VAR:-default}` for defaults +- `[[ ]]` over `[ ]` +- Quote all paths: `"${directory}/${filename}"` + +## Argument Parsing + +- Provide `usage()` — list parameters, flags, env vars, copy-pastable examples with expected output +- Support at minimum `--help` (print `usage()`, exit 0) +- Support `--dry-run` for destructive or irreversible operations + +## Dependency Checks + +- Check required tools early: `command -v || { echo "Install : "; exit 1; }` + +## Portability + +- Prefer POSIX idioms; guard GNU vs BSD differences explicitly +- `sed -i` requires empty suffix on macOS: `sed -i ''` vs Linux `sed -i` +- `date -d` GNU only — use `date -v` on macOS or detect OS and branch + +## Error Handling and Exit Codes + +- Trap `ERR`/`EXIT` for cleanup and meaningful error messages +- Return meaningful non-zero exit codes — document semantics in `usage()` + +## Security + +- No `eval` — use arrays, named variables, or functions +- Sanitize external inputs before use in commands or filenames +- Never log or write secrets to temp files +- Use `mktemp` for temp files; restrict permissions with `chmod 600` or `umask 077` +- Clean up temp files in `trap ... EXIT` + +## Linting, Formatting, and CI + +- Verify all scripts with `shellcheck`; run in CI +- Format with `shfmt`; run in CI +- Document `shellcheck` exceptions inline with justification: `# shellcheck disable=SC2xxx — reason` +- Test scripts: name with prefix `test` (e.g. `testMyFeature.sh`) — `runTests.sh` auto-discovers in `scripts/` and `domains/` dirs diff --git a/.github/skills/caveman/LICENSE b/.github/skills/caveman/LICENSE new file mode 100644 index 000000000..6ad57f8ef --- /dev/null +++ b/.github/skills/caveman/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Julius Brussee + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. \ No newline at end of file diff --git a/.github/skills/caveman/SKILL.md b/.github/skills/caveman/SKILL.md new file mode 100644 index 000000000..39b5ab812 --- /dev/null +++ b/.github/skills/caveman/SKILL.md @@ -0,0 +1,58 @@ +--- +name: caveman +description: > + Ultra-compressed communication mode. Cuts token usage ~75% by speaking like caveman + while keeping full technical accuracy. Supports intensity levels: lite, full (default), ultra. + Use when user says "caveman mode", "talk like caveman", "use caveman", "less tokens", + "be brief", or invokes /caveman. Also auto-triggers when token efficiency is requested. +--- + +Respond terse like smart caveman. All technical substance stay. Only fluff die. + +## Persistence + +ACTIVE EVERY RESPONSE. No revert after many turns. No filler drift. Still active if unsure. Off only: "stop caveman" / "normal mode". + +Default: **full**. Switch: `/caveman lite|full|ultra`. + +## Rules + +Drop: articles (a/an/the), filler (just/really/basically/actually/simply), pleasantries (sure/certainly/of course/happy to), hedging. Fragments OK. Short synonyms (big not extensive, fix not "implement a solution for"). Technical terms exact. Code blocks unchanged. Errors quoted exact. + +Pattern: `[thing] [action] [reason]. [next step].` + +Not: "Sure! I'd be happy to help you with that. The issue you're experiencing is likely caused by..." +Yes: "Bug in auth middleware. Token expiry check use `<` not `<=`. Fix:" + +## Intensity + +| Level | What change | +|-------|------------| +| **lite** | No filler/hedging. Keep articles + full sentences. Professional but tight | +| **full** | Drop articles, fragments OK, short synonyms. Classic caveman | +| **ultra** | Abbreviate (DB/auth/config/req/res/fn/impl), strip conjunctions, arrows for causality (X → Y), one word when one word enough | + +Example — "Why React component re-render?" +- lite: "Your component re-renders because you create a new object reference each render. Wrap it in `useMemo`." +- full: "New object ref each render. Inline object prop = new ref = re-render. Wrap in `useMemo`." +- ultra: "Inline obj prop → new ref → re-render. `useMemo`." + +Example — "Explain database connection pooling." +- lite: "Connection pooling reuses open connections instead of creating new ones per request. Avoids repeated handshake overhead." +- full: "Pool reuse open DB connections. No new connection per request. Skip handshake overhead." +- ultra: "Pool = reuse DB conn. Skip handshake → fast under load." + +## Auto-Clarity + +Drop caveman for: security warnings, irreversible action confirmations, multi-step sequences where fragment order risks misread, user asks to clarify or repeats question. Resume caveman after clear part done. + +Example — destructive op: +> **Warning:** This will permanently delete all rows in the `users` table and cannot be undone. +> ```sql +> DROP TABLE users; +> ``` +> Caveman resume. Verify backup exist first. + +## Boundaries + +Code/commits/PRs: write normal. "stop caveman" or "normal mode": revert. Level persist until changed or session end. \ No newline at end of file diff --git a/.github/skills/neo4j-management/SKILL.md b/.github/skills/neo4j-management/SKILL.md index 59bc5e895..0a7dd88b9 100644 --- a/.github/skills/neo4j-management/SKILL.md +++ b/.github/skills/neo4j-management/SKILL.md @@ -16,7 +16,7 @@ Manage the local Neo4j graph database lifecycle: detect, start, stop, reconfigur | Start | `startNeo4j.sh` | Neo4j not running | | Stop | `stopNeo4j.sh` | Free resources, switch workspace | | Memory | `useNeo4jHighMemoryProfile.sh` | Out-of-memory or slow queries | -| Setup | Use `analyze.sh` (see [GETTING_STARTED](../../GETTING_STARTED.md)) | New installation | +| Setup | Use `analyze.sh` (see [GETTING_STARTED](./../../../GETTING_STARTED.md)) | New installation | ## When to Use @@ -145,7 +145,7 @@ cd ### Step 3 — Setup / Installation Requests -> **Always recommend using `analyze.sh` first.** It handles download, installation, configuration, start, and stop automatically via profiles. Point the user to the [GETTING_STARTED guide](../../GETTING_STARTED.md). +> **Always recommend using `analyze.sh` first.** It handles download, installation, configuration, start, and stop automatically via profiles. Point the user to the [GETTING_STARTED guide](./../../../GETTING_STARTED.md). Only assist with a manual `setupNeo4j.sh` invocation if the user can explain a concrete reason why the automated `analyze.sh` workflow does not fit their needs (e.g. custom Neo4j version, enterprise edition, non-standard directory layout). diff --git a/.github/skills/start-new-analysis/SKILL.md b/.github/skills/start-new-analysis/SKILL.md new file mode 100644 index 000000000..4b048152e --- /dev/null +++ b/.github/skills/start-new-analysis/SKILL.md @@ -0,0 +1,207 @@ +--- +name: start-new-analysis +description: Use when user wants to start a new code analysis, analyze Java or TypeScript project, set up a new analysis workspace, or run code-graph-analysis-pipeline for the first time on a project. +--- + +# Start New Analysis + +Sets up, runs new analysis project end-to-end. + +**Prereq:** Run all commands from repo root (`code-graph-analysis-pipeline/`). Requires `NEO4J_INITIAL_PASSWORD` set. + +--- + +## Step 1 — Choose Path + +Ask: "Analyze pre-built example or custom project?" + +| Option | Project | +|--------|---------| +| AxonFramework | Java event-sourcing library | +| ReactRouter | TypeScript UI routing library | +| Custom | Any Java or TypeScript project | + +If **AxonFramework** or **ReactRouter**: skip Steps 4–6, go straight to Step 2 → 3 → [Example Script](#example-scripts). +If **Custom**: follow all steps. + +--- + +## Step 2 — Neo4j Password + +Ask: "Local run on own machine? If yes, I can set `NEO4J_INITIAL_PASSWORD` — provide password. Otherwise, run `export NEO4J_INITIAL_PASSWORD=` in terminal and confirm when done." + +If user provides password: +```shell +export NEO4J_INITIAL_PASSWORD= +``` + +> **Security warning:** Only accept the password directly in agent context when the user explicitly confirms this is a local run on their own machine. Never log, echo, or display the password value after setting it. + +--- + +## Step 3 — Neo4j State + +Run from repo root: +```shell +./domains/neo4j-management/detectNeo4j.sh +``` + +| Result | Action | +|--------|--------| +| `Neo4j not running` | proceed | +| `Neo4j running in ` | **warn**: data from different projects will mix. Ask user to stop it: `cd && ./stopNeo4j.sh`. Only continue after confirmed stopped. | + +--- + +## Step 3b — Check Compatibility + Advise Report Type + +Run from repo root: +```shell +./scripts/checkCompatibility.sh +``` + +Read output and advise report type: + +| Output | Recommended flag | +|--------|-----------------| +| ❌ on required deps (java, git, curl, jq…) | Fix missing deps first — analysis will fail | +| ✅ required, but ❌/⚠️ on Python env (conda/venv) or python/pip | `--report Csv` | +| ✅ required + ✅ Python env + ✅ python/pip | `--report Python` (no PDF) or full (no flag) | +| ✅ all including jupyter | `--report Jupyter` (adds PDF, needs Chromium) | +| ✅ npx/npm or dot | add `--report Visualization` or run full | + +Tell user which `--report` flag to recommend and why. Confirm before proceeding. + +--- + +## Example Scripts + +For **AxonFramework** — ask for optional version (omit to use latest): +```shell +./scripts/examples/analyzeAxonFramework.sh [] --report +``` +Example: `./scripts/examples/analyzeAxonFramework.sh 5.0.3 --report Csv` + +For **ReactRouter** — ask for optional version (omit to use latest): +```shell +./scripts/examples/analyzeReactRouter.sh [] --report +``` +Example: `./scripts/examples/analyzeReactRouter.sh 7.5.0 --report Csv` + +Both scripts: auto-detect latest version if omitted, call `init.sh` internally, download artifacts, then run `analyze.sh`. No further steps needed — skip to [Step 7 notes](#step-7--start-analysis). + +--- + +## Step 4 — Initialize (Custom Only) + +From **repo root**: +```shell +./init.sh +cd temp/ +``` + +Creates `artifacts/`, `source/`, forwarding scripts (`analyze.sh`, `startNeo4j.sh`, `stopNeo4j.sh`) in workspace. +Fails if `NEO4J_INITIAL_PASSWORD` not set. + +--- + +## Step 5 — Add Code to Analyze (Custom Only) + +Ask: "Project Java, TypeScript, or both?" + +### Java + +Ask: "Copy manually, symlink, or download from Maven Central?" + +Option A — Copy/move manually: +```shell +cp /path/to/*.jar temp//artifacts/ +``` + +Option B — Symlink: +```shell +ln -s /absolute/path/to/file.jar temp//artifacts/ +``` + +Option C — Download from Maven Central (from `temp/`): +```shell +./../../scripts/downloadMavenArtifact.sh -g -a -v [-t jar] +``` +Example: `-g org.axonframework -a axon-messaging -v 4.10.0` + +### TypeScript + +Option A — Download via helper (clone + dep install, from `temp/`): +```shell +./../../scripts/downloader/downloadTypescriptProject.sh \ + --url \ + --version \ + [--tag ] \ + [--packageManager npm|pnpm|yarn] +``` + +Option B — Clone manually into `source/` (from `temp/`): +```shell +git clone ./source/ +# shallow clone (no git history): +git clone --depth 1 ./source/ +# symlink: +ln -s /absolute/path/to/project ./source/ +``` + +Then install JS/TS dependencies: +```shell +cd source/ && npm install # or: pnpm install / yarn install +cd ../.. +``` + +--- + +## Step 6 — Git History (Custom Only, Optional) + +Ask: "Include git history analysis (change frequency, author stats)?" + +If yes: git repo must be in `source/`. If not already cloned (from `temp/`): +```shell +# Full history: +git clone ./source/ +# Bare clone (Java only, no source needed): +git clone --bare ./source/.git +``` + +Default import mode: `plugin` (jQAssistant handles automatically — recommended). +Override with env var if needed: + +| `IMPORT_GIT_LOG_DATA_IF_SOURCE_IS_PRESENT` | Effect | +|--------------------------------------------|--------| +| `plugin` | default, recommended | +| `none` | skip git import | +| `aggregated` | monthly summary, smaller dataset | +| `full` | all commits | + +```shell +export IMPORT_GIT_LOG_DATA_IF_SOURCE_IS_PRESENT=none # example: disable +``` + +--- + +## Step 7 — Start Analysis + +Use recommended `--report` flag from Step 3b. Run from `temp/`: +```shell +./analyze.sh --report Csv +``` + +| Flag | Requires | Use when | +|------|----------|----------| +| `--report Csv` | nothing extra | data only, fastest | +| `--report Python` | Python + Conda/venv | charts, no PDF | +| `--report Jupyter` | + Chromium | charts + PDF | +| `--report Visualization` | GraphViz or npx | graph images | +| `--explore` | nothing | manual Neo4j exploration only | + +After analysis: `http://localhost:7474/browser` — user: `neo4j`, password from Step 2. + +**Optional flags:** +- `--keep-running` — Neo4j stays up after analysis (useful for rerunning reports) +- `--domain ` — single domain only (e.g. `anomaly-detection`, `git-history`) diff --git a/.github/skills/writing-plans/LICENSE b/.github/skills/writing-plans/LICENSE new file mode 100644 index 000000000..abf039032 --- /dev/null +++ b/.github/skills/writing-plans/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2025 Jesse Vincent + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/.github/skills/writing-plans/SKILL.md b/.github/skills/writing-plans/SKILL.md new file mode 100644 index 000000000..e595a6fe4 --- /dev/null +++ b/.github/skills/writing-plans/SKILL.md @@ -0,0 +1,152 @@ +--- +name: writing-plans +description: Use when you have a spec or requirements for a multi-step task, before touching code +--- + +# Writing Plans + +## Overview + +Write implementation plans assuming engineer has zero context for codebase and questionable taste. Document everything needed: files to touch per task, code, testing, docs to check, how to test. Give complete plan as bite-sized tasks. DRY. YAGNI. TDD. Frequent commits. + +Assume skilled developer, knows almost nothing about toolset or problem domain. Assume poor test design knowledge. + +**Announce at start:** "I'm using the writing-plans skill to create the implementation plan." + +**Context:** Run in dedicated worktree (created by brainstorming skill). + +**Save plans to:** `docs/superpowers/plans/YYYY-MM-DD-.md` +- (User preferences for plan location override this default) + +## Scope Check + +If spec covers multiple independent subsystems, should have been broken into sub-project specs during brainstorming. If not, suggest separate plans — one per subsystem. Each plan produces working, testable software on its own. + +## File Structure + +Before defining tasks, map out files to create or modify and what each is responsible for. Decomposition decisions get locked in here. + +- Design units with clear boundaries and well-defined interfaces. Each file has one clear responsibility. +- Edits more reliable when files focused. Prefer smaller, focused files over large ones that do too much. +- Files that change together live together. Split by responsibility, not by technical layer. +- In existing codebases, follow established patterns. If codebase uses large files, don't unilaterally restructure — but if file you're modifying grown unwieldy, including split in plan is reasonable. + +Structure informs task decomposition. Each task produces self-contained changes that make sense independently. + +## Bite-Sized Task Granularity + +**Each step is one action (2-5 minutes):** +- "Write the failing test" - step +- "Run it to make sure it fails" - step +- "Implement the minimal code to make the test pass" - step +- "Run the tests and make sure they pass" - step +- "Commit" - step + +## Plan Document Header + +**Every plan MUST start with this header:** + +```markdown +# [Feature Name] Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** [One sentence describing what this builds] + +**Architecture:** [2-3 sentences about approach] + +**Tech Stack:** [Key technologies/libraries] + +--- +``` + +## Task Structure + +````markdown +### Task N: [Component Name] + +**Files:** +- Create: `exact/path/to/file.py` +- Modify: `exact/path/to/existing.py:123-145` +- Test: `tests/exact/path/to/test.py` + +- [ ] **Step 1: Write the failing test** + +```python +def test_specific_behavior(): + result = function(input) + assert result == expected +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `pytest tests/path/test.py::test_name -v` +Expected: FAIL with "function not defined" + +- [ ] **Step 3: Write minimal implementation** + +```python +def function(input): + return expected +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `pytest tests/path/test.py::test_name -v` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add tests/path/test.py src/path/file.py +git commit -m "feat: add specific feature" +``` +```` + +## No Placeholders + +Every step needs actual content engineer can use. These are **plan failures** — never write them: +- "TBD", "TODO", "implement later", "fill in details" +- "Add appropriate error handling" / "add validation" / "handle edge cases" +- "Write tests for the above" (without actual test code) +- "Similar to Task N" (repeat the code — the engineer may be reading tasks out of order) +- Steps that describe what to do without showing how (code blocks required for code steps) +- References to types, functions, or methods not defined in any task + +## Remember +- Exact file paths always +- Complete code in every step — if step changes code, show code +- Exact commands with expected output +- DRY, YAGNI, TDD, frequent commits + +## Self-Review + +After writing complete plan, check plan against spec. Checklist to run yourself — not subagent dispatch. + +**1. Spec coverage:** Skim each section/requirement in spec. Can you point to task implementing it? List gaps. + +**2. Placeholder scan:** Search plan for red flags — patterns from "No Placeholders" section. Fix them. + +**3. Type consistency:** Do types, method signatures, property names in later tasks match earlier tasks? Function called `clearLayers()` in Task 3 but `clearFullLayers()` in Task 7 is bug. + +If issues found, fix inline. No re-review needed — fix and move on. If spec requirement has no task, add task. + +## Execution Handoff + +After saving plan, offer execution choice: + +**"Plan complete and saved to `docs/superpowers/plans/.md`. Two execution options:** + +**1. Subagent-Driven (recommended)** - Dispatch fresh subagent per task, review between tasks, fast iteration + +**2. Inline Execution** - Execute tasks in session using executing-plans, batch execution with checkpoints + +**Which approach?"** + +**If Subagent-Driven chosen:** +- **REQUIRED SUB-SKILL:** Use superpowers:subagent-driven-development +- Fresh subagent per task + two-stage review + +**If Inline Execution chosen:** +- **REQUIRED SUB-SKILL:** Use superpowers:executing-plans +- Batch execution with checkpoints for review diff --git a/.github/skills/writing-plans/plan-document-reviewer-prompt.md b/.github/skills/writing-plans/plan-document-reviewer-prompt.md new file mode 100644 index 000000000..7166a664c --- /dev/null +++ b/.github/skills/writing-plans/plan-document-reviewer-prompt.md @@ -0,0 +1,49 @@ +# Plan Document Reviewer Prompt Template + +Use when dispatching plan document reviewer subagent. + +**Purpose:** Verify plan complete, matches spec, has proper task decomposition. + +**Dispatch after:** Complete plan written. + +``` +Task tool (general-purpose): + description: "Review plan document" + prompt: | + You are a plan document reviewer. Verify this plan is complete and ready for implementation. + + **Plan to review:** [PLAN_FILE_PATH] + **Spec for reference:** [SPEC_FILE_PATH] + + ## What to Check + + | Category | What to Look For | + |----------|------------------| + | Completeness | TODOs, placeholders, incomplete tasks, missing steps | + | Spec Alignment | Plan covers spec requirements, no major scope creep | + | Task Decomposition | Tasks have clear boundaries, steps are actionable | + | Buildability | Could an engineer follow this plan without getting stuck? | + + ## Calibration + + **Only flag issues that would cause real problems during implementation.** + An implementer building the wrong thing or getting stuck is an issue. + Minor wording, stylistic preferences, and "nice to have" suggestions are not. + + Approve unless there are serious gaps — missing requirements from the spec, + contradictory steps, placeholder content, or tasks so vague they can't be acted on. + + ## Output Format + + ## Plan Review + + **Status:** Approved | Issues Found + + **Issues (if any):** + - [Task X, Step Y]: [specific issue] - [why it matters for implementation] + + **Recommendations (advisory, do not block approval):** + - [suggestions for improvement] +``` + +**Reviewer returns:** Status, Issues (if any), Recommendations diff --git a/.github/skills/writing-skills/LICENSE b/.github/skills/writing-skills/LICENSE new file mode 100644 index 000000000..abf039032 --- /dev/null +++ b/.github/skills/writing-skills/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2025 Jesse Vincent + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/.github/skills/writing-skills/SKILL.md b/.github/skills/writing-skills/SKILL.md new file mode 100644 index 000000000..7585a2063 --- /dev/null +++ b/.github/skills/writing-skills/SKILL.md @@ -0,0 +1,655 @@ +--- +name: writing-skills +description: Use when creating new skills, editing existing skills, or verifying skills work before deployment +--- + +# Writing Skills + +## Overview + +**Writing skills IS Test-Driven Development applied to process documentation.** + +**Personal skills live in agent-specific directories (`~/.claude/skills` for Claude Code, `~/.agents/skills/` for Codex)** + +Write test cases (pressure scenarios with subagents), watch fail (baseline behavior), write skill (documentation), watch tests pass (agents comply), refactor (close loopholes). + +**Core principle:** Didn't watch agent fail without skill → don't know if skill teaches right thing. + +**REQUIRED BACKGROUND:** MUST understand superpowers:test-driven-development before using skill. That skill defines RED-GREEN-REFACTOR cycle. This skill adapts TDD to documentation. + +**Official guidance:** See anthropic-best-practices.md for Anthropic's official skill authoring best practices. Provides additional patterns complementing TDD-focused approach in this skill. + +## What Is Skill? + +**Skill** = reference guide for proven techniques, patterns, tools. Skills help future Claude instances find and apply effective approaches. + +**Skills are:** Reusable techniques, patterns, tools, reference guides + +**Skills are NOT:** Narratives about how you solved problem once + +## TDD Mapping for Skills + +| TDD Concept | Skill Creation | +|-------------|----------------| +| **Test case** | Pressure scenario with subagent | +| **Production code** | Skill document (SKILL.md) | +| **Test fails (RED)** | Agent violates rule without skill (baseline) | +| **Test passes (GREEN)** | Agent complies with skill present | +| **Refactor** | Close loopholes while maintaining compliance | +| **Write test first** | Run baseline scenario BEFORE writing skill | +| **Watch it fail** | Document exact rationalizations agent uses | +| **Minimal code** | Write skill addressing those specific violations | +| **Watch it pass** | Verify agent now complies | +| **Refactor cycle** | Find new rationalizations → plug → re-verify | + +The entire skill creation process follows RED-GREEN-REFACTOR. + +## When to Create Skill + +**Create when:** +- Technique wasn't intuitively obvious to you +- You'd reference this again across projects +- Pattern applies broadly (not project-specific) +- Others would benefit + +**Don't create for:** +- One-off solutions +- Standard practices well-documented elsewhere +- Project-specific conventions (put in CLAUDE.md) +- Mechanical constraints (if it's enforceable with regex/validation, automate it—save documentation for judgment calls) + +## Skill Types + +### Technique +Concrete method with steps to follow (condition-based-waiting, root-cause-tracing) + +### Pattern +Way of thinking about problems (flatten-with-flags, test-invariants) + +### Reference +API docs, syntax guides, tool documentation (office docs) + +## Directory Structure + + +``` +skills/ + skill-name/ + SKILL.md # Main reference (required) + supporting-file.* # Only if needed +``` + +**Flat namespace** - all skills in one searchable namespace + +**Separate files for:** +1. **Heavy reference** (100+ lines) - API docs, comprehensive syntax +2. **Reusable tools** - Scripts, utilities, templates + +**Keep inline:** +- Principles and concepts +- Code patterns (< 50 lines) +- Everything else + +## SKILL.md Structure + +**Frontmatter (YAML):** +- Two required fields: `name` and `description` (see [agentskills.io/specification](https://agentskills.io/specification) for all supported fields) +- Max 1024 characters total +- `name`: Use letters, numbers, hyphens only (no parentheses, special chars) + - `description`: Third-person, describes ONLY when to use (NOT what it does) + - Start with "Use when..." to focus on triggering conditions + - Include specific symptoms, situations, contexts + - **NEVER summarize skill's process or workflow** (see CSO section for why) + - Keep under 500 characters if possible + +```markdown +--- +name: Skill-Name-With-Hyphens +description: Use when [specific triggering conditions and symptoms] +--- + +# Skill Name + +## Overview +What is this? Core principle in 1-2 sentences. + +## When to Use +[Small inline flowchart IF decision non-obvious] + +Bullet list with SYMPTOMS and use cases +When NOT to use + +## Core Pattern (for techniques/patterns) +Before/after code comparison + +## Quick Reference +Table or bullets for scanning common operations + +## Implementation +Inline code for simple patterns +Link to file for heavy reference or reusable tools + +## Common Mistakes +What goes wrong + fixes + +## Real-World Impact (optional) +Concrete results +``` + + +## Claude Search Optimization (CSO) + +**Critical for discovery:** Future Claude needs to FIND skill + +### 1. Rich Description Field + +**Purpose:** Claude reads description to decide which skills to load for given task. Make it answer: "Should I read this skill right now?" + +**Format:** Start with "Use when..." to focus on triggering conditions + +**CRITICAL: Description = When to Use, NOT What Skill Does** + +Description ONLY describes triggering conditions. Do NOT summarize skill's process or workflow. + +**Why this matters:** Testing revealed when description summarizes skill's workflow, Claude may follow description instead of reading full skill content. Description "code review between tasks" caused Claude to do ONE review, even though skill's flowchart showed TWO reviews (spec compliance then code quality). + +When description changed to "Use when executing implementation plans with independent tasks" (no workflow summary), Claude correctly read flowchart and followed two-stage review process. + +**Trap:** Descriptions summarizing workflow create shortcut Claude will take. Skill body becomes documentation Claude skips. + +```yaml +# ❌ BAD: Summarizes workflow - Claude may follow this instead of reading skill +description: Use when executing plans - dispatches subagent per task with code review between tasks + +# ❌ BAD: Too much process detail +description: Use for TDD - write test first, watch it fail, write minimal code, refactor + +# ✅ GOOD: Just triggering conditions, no workflow summary +description: Use when executing implementation plans with independent tasks in the current session + +# ✅ GOOD: Triggering conditions only +description: Use when implementing any feature or bugfix, before writing implementation code +``` + +**Content:** +- Use concrete triggers, symptoms, situations signaling skill applies +- Describe *problem* (race conditions, inconsistent behavior) not *language-specific symptoms* (setTimeout, sleep) +- Keep triggers technology-agnostic unless skill itself is technology-specific +- If skill is technology-specific, make explicit in trigger +- Write in third person (injected into system prompt) +- **NEVER summarize skill's process or workflow** + +```yaml +# ❌ BAD: Too abstract, vague, doesn't include when to use +description: For async testing + +# ❌ BAD: First person +description: I can help you with async tests when they're flaky + +# ❌ BAD: Mentions technology but skill isn't specific to it +description: Use when tests use setTimeout/sleep and are flaky + +# ✅ GOOD: Starts with "Use when", describes problem, no workflow +description: Use when tests have race conditions, timing dependencies, or pass/fail inconsistently + +# ✅ GOOD: Technology-specific skill with explicit trigger +description: Use when using React Router and handling authentication redirects +``` + +### 2. Keyword Coverage + +Use words Claude would search for: +- Error messages: "Hook timed out", "ENOTEMPTY", "race condition" +- Symptoms: "flaky", "hanging", "zombie", "pollution" +- Synonyms: "timeout/hang/freeze", "cleanup/teardown/afterEach" +- Tools: Actual commands, library names, file types + +### 3. Descriptive Naming + +**Use active voice, verb-first:** +- ✅ `creating-skills` not `skill-creation` +- ✅ `condition-based-waiting` not `async-test-helpers` + +### 4. Token Efficiency (Critical) + +**Problem:** getting-started and frequently-referenced skills load into EVERY conversation. Every token counts. + +**Target word counts:** +- getting-started workflows: <150 words each +- Frequently-loaded skills: <200 words total +- Other skills: <500 words (still be concise) + +**Techniques:** + +**Move details to tool help:** +```bash +# ❌ BAD: Document all flags in SKILL.md +search-conversations supports --text, --both, --after DATE, --before DATE, --limit N + +# ✅ GOOD: Reference --help +search-conversations supports multiple modes and filters. Run --help for details. +``` + +**Use cross-references:** +```markdown +# ❌ BAD: Repeat workflow details +When searching, dispatch subagent with template... +[20 lines of repeated instructions] + +# ✅ GOOD: Reference other skill +Always use subagents (50-100x context savings). REQUIRED: Use [other-skill-name] for workflow. +``` + +**Compress examples:** +```markdown +# ❌ BAD: Verbose example (42 words) +your human partner: "How did we handle authentication errors in React Router before?" +You: I'll search past conversations for React Router authentication patterns. +[Dispatch subagent with search query: "React Router authentication error handling 401"] + +# ✅ GOOD: Minimal example (20 words) +Partner: "How did we handle auth errors in React Router?" +You: Searching... +[Dispatch subagent → synthesis] +``` + +**Eliminate redundancy:** +- Don't repeat what's in cross-referenced skills +- Don't explain what's obvious from command +- Don't include multiple examples of same pattern + +**Verification:** +```bash +wc -w skills/path/SKILL.md +# getting-started workflows: aim for <150 each +# Other frequently-loaded: aim for <200 total +``` + +**Name by what you DO or core insight:** +- ✅ `condition-based-waiting` > `async-test-helpers` +- ✅ `using-skills` not `skill-usage` +- ✅ `flatten-with-flags` > `data-structure-refactoring` +- ✅ `root-cause-tracing` > `debugging-techniques` + +**Gerunds (-ing) work well for processes:** +- `creating-skills`, `testing-skills`, `debugging-with-logs` +- Active, describes action being taken + +### 4. Cross-Referencing Other Skills + +**When writing documentation referencing other skills:** + +Use skill name only, with explicit requirement markers: +- ✅ Good: `**REQUIRED SUB-SKILL:** Use superpowers:test-driven-development` +- ✅ Good: `**REQUIRED BACKGROUND:** You MUST understand superpowers:systematic-debugging` +- ❌ Bad: `See skills/testing/test-driven-development` (unclear if required) +- ❌ Bad: `@skills/testing/test-driven-development/SKILL.md` (force-loads, burns context) + +**Why no @ links:** `@` syntax force-loads files immediately, consuming 200k+ context before needed. + +## Flowchart Usage + +```dot +digraph when_flowchart { + "Need to show information?" [shape=diamond]; + "Decision where I might go wrong?" [shape=diamond]; + "Use markdown" [shape=box]; + "Small inline flowchart" [shape=box]; + + "Need to show information?" -> "Decision where I might go wrong?" [label="yes"]; + "Decision where I might go wrong?" -> "Small inline flowchart" [label="yes"]; + "Decision where I might go wrong?" -> "Use markdown" [label="no"]; +} +``` + +**Use flowcharts ONLY for:** +- Non-obvious decision points +- Process loops where you might stop too early +- "When to use A vs B" decisions + +**Never use flowcharts for:** +- Reference material → Tables, lists +- Code examples → Markdown blocks +- Linear instructions → Numbered lists +- Labels without semantic meaning (step1, helper2) + +See @graphviz-conventions.dot for graphviz style rules. + +**Visualizing for human partner:** Use `render-graphs.js` in this directory to render skill's flowcharts to SVG: +```bash +./render-graphs.js ../some-skill # Each diagram separately +./render-graphs.js ../some-skill --combine # All diagrams in one SVG +``` + +## Code Examples + +**One excellent example beats many mediocre ones** + +Choose most relevant language: +- Testing techniques → TypeScript/JavaScript +- System debugging → Shell/Python +- Data processing → Python + +**Good example:** +- Complete and runnable +- Well-commented explaining WHY +- From real scenario +- Shows pattern clearly +- Ready to adapt (not generic template) + +**Don't:** +- Implement in 5+ languages +- Create fill-in-the-blank templates +- Write contrived examples + +Good at porting — one great example is enough. + +## File Organization + +### Self-Contained Skill +``` +defense-in-depth/ + SKILL.md # Everything inline +``` +When: All content fits, no heavy reference needed + +### Skill with Reusable Tool +``` +condition-based-waiting/ + SKILL.md # Overview + patterns + example.ts # Working helpers to adapt +``` +When: Tool is reusable code, not only narrative + +### Skill with Heavy Reference +``` +pptx/ + SKILL.md # Overview + workflows + pptxgenjs.md # 600 lines API reference + ooxml.md # 500 lines XML structure + scripts/ # Executable tools +``` +When: Reference material too large for inline + +## The Iron Law (Same as TDD) + +``` +NO SKILL WITHOUT A FAILING TEST FIRST +``` + +Applies to NEW skills AND EDITS to existing skills. + +Write skill before testing? Delete it. Start over. +Edit skill without testing? Same violation. + +**No exceptions:** +- Not for "simple additions" +- Not for "just adding a section" +- Not for "documentation updates" +- Don't keep untested changes as "reference" +- Don't "adapt" while running tests +- Delete means delete + +**REQUIRED BACKGROUND:** superpowers:test-driven-development skill explains why this matters. Same principles apply to documentation. + +## Testing All Skill Types + +Skill types need different test approaches: + +### Discipline-Enforcing Skills (rules/requirements) + +**Examples:** TDD, verification-before-completion, designing-before-coding + +**Test with:** +- Academic questions: Do they understand rules? +- Pressure scenarios: Do they comply under stress? +- Multiple pressures combined: time + sunk cost + exhaustion +- Identify rationalizations and add explicit counters + +**Success criteria:** Agent follows rule under maximum pressure + +### Technique Skills (how-to guides) + +**Examples:** condition-based-waiting, root-cause-tracing, defensive-programming + +**Test with:** +- Application scenarios: Can they apply technique correctly? +- Variation scenarios: Do they handle edge cases? +- Missing information tests: Do instructions have gaps? + +**Success criteria:** Agent successfully applies technique to new scenario + +### Pattern Skills (mental models) + +**Examples:** reducing-complexity, information-hiding concepts + +**Test with:** +- Recognition scenarios: Do they recognize when pattern applies? +- Application scenarios: Can they use mental model? +- Counter-examples: Do they know when NOT to apply? + +**Success criteria:** Agent correctly identifies when/how to apply pattern + +### Reference Skills (documentation/APIs) + +**Examples:** API documentation, command references, library guides + +**Test with:** +- Retrieval scenarios: Can they find right information? +- Application scenarios: Can they use what they found correctly? +- Gap testing: Are common use cases covered? + +**Success criteria:** Agent finds and correctly applies reference information + +## Common Rationalizations for Skipping Testing + +| Excuse | Reality | +|--------|---------| +| "Skill is obviously clear" | Clear to you ≠ clear to other agents. Test it. | +| "It's just a reference" | References can have gaps, unclear sections. Test retrieval. | +| "Testing is overkill" | Untested skills have issues. Always. 15 min testing saves hours. | +| "I'll test if problems emerge" | Problems = agents can't use skill. Test BEFORE deploying. | +| "Too tedious to test" | Testing is less tedious than debugging bad skill in production. | +| "I'm confident it's good" | Overconfidence guarantees issues. Test anyway. | +| "Academic review is enough" | Reading ≠ using. Test application scenarios. | +| "No time to test" | Deploying untested skill wastes more time fixing it later. | + +**All of these mean: Test before deploying. No exceptions.** + +## Bulletproofing Skills Against Rationalization + +Skills enforcing discipline (like TDD) need to resist rationalization. Agents are smart, find loopholes under pressure. + +**Psychology note:** Understanding WHY persuasion techniques work helps apply systematically. See persuasion-principles.md for research foundation (Cialdini, 2021; Meincke et al., 2025) on authority, commitment, scarcity, social proof, unity. + +### Close Every Loophole Explicitly + +Don't state rule only — forbid specific workarounds: + + +```markdown +Write code before test? Delete it. +``` + + + +```markdown +Write code before test? Delete it. Start over. + +**No exceptions:** +- Don't keep it as "reference" +- Don't "adapt" it while writing tests +- Don't look at it +- Delete means delete +``` + + +### Address "Spirit vs Letter" Arguments + +Add foundational principle early: + +```markdown +**Violating the letter of the rules is violating the spirit of the rules.** +``` + +Cuts off entire class of "I'm following spirit" rationalizations. + +### Build Rationalization Table + +Capture rationalizations from baseline testing (see Testing section below). Every excuse agents make goes in table: + +```markdown +| Excuse | Reality | +|--------|---------| +| "Too simple to test" | Simple code breaks. Test takes 30 seconds. | +| "I'll test after" | Tests passing immediately prove nothing. | +| "Tests after achieve same goals" | Tests-after = "what does this do?" Tests-first = "what should this do?" | +``` + +### Create Red Flags List + +Make it easy for agents to self-check when rationalizing: + +```markdown +## Red Flags - STOP and Start Over + +- Code before test +- "I already manually tested it" +- "Tests after achieve the same purpose" +- "It's about spirit not ritual" +- "This is different because..." + +**All of these mean: Delete code. Start over with TDD.** +``` + +### Update CSO for Violation Symptoms + +Add to description: symptoms of when ABOUT to violate rule: + +```yaml +description: use when implementing any feature or bugfix, before writing implementation code +``` + +## RED-GREEN-REFACTOR for Skills + +Follow TDD cycle: + +### RED: Write Failing Test (Baseline) + +Run pressure scenario with subagent WITHOUT skill. Document exact behavior: +- What choices did they make? +- What rationalizations did they use (verbatim)? +- Which pressures triggered violations? + +This is "watch test fail" — must see what agents naturally do before writing skill. + +### GREEN: Write Minimal Skill + +Write skill addressing those specific rationalizations. Don't add extra content for hypothetical cases. + +Run same scenarios WITH skill. Agent should now comply. + +### REFACTOR: Close Loopholes + +Agent found new rationalization? Add explicit counter. Re-test until bulletproof. + +**Testing methodology:** See @testing-skills-with-subagents.md for complete testing methodology: +- How to write pressure scenarios +- Pressure types (time, sunk cost, authority, exhaustion) +- Plugging holes systematically +- Meta-testing techniques + +## Anti-Patterns + +### ❌ Narrative Example +"In session 2025-10-03, we found empty projectDir caused..." +**Why bad:** Too specific, not reusable + +### ❌ Multi-Language Dilution +example-js.js, example-py.py, example-go.go +**Why bad:** Mediocre quality, maintenance burden + +### ❌ Code in Flowcharts +```dot +step1 [label="import fs"]; +step2 [label="read file"]; +``` +**Why bad:** Can't copy-paste, hard to read + +### ❌ Generic Labels +helper1, helper2, step3, pattern4 +**Why bad:** Labels should have semantic meaning + +## STOP: Before Moving to Next Skill + +**After writing ANY skill, you MUST STOP and complete deployment process.** + +**Do NOT:** +- Create multiple skills in batch without testing each +- Move to next skill before current one is verified +- Skip testing because "batching is more efficient" + +**The deployment checklist below is MANDATORY for EACH skill.** + +Deploying untested skills = deploying untested code. Violation of quality standards. + +## Skill Creation Checklist (TDD Adapted) + +**IMPORTANT: Use TodoWrite to create todos for EACH checklist item below.** + +**RED Phase - Write Failing Test:** +- [ ] Create pressure scenarios (3+ combined pressures for discipline skills) +- [ ] Run scenarios WITHOUT skill - document baseline behavior verbatim +- [ ] Identify patterns in rationalizations/failures + +**GREEN Phase - Write Minimal Skill:** +- [ ] Name uses only letters, numbers, hyphens (no parentheses/special chars) +- [ ] YAML frontmatter with required `name` and `description` fields (max 1024 chars; see [spec](https://agentskills.io/specification)) +- [ ] Description starts with "Use when..." and includes specific triggers/symptoms +- [ ] Description written in third person +- [ ] Keywords throughout for search (errors, symptoms, tools) +- [ ] Clear overview with core principle +- [ ] Address specific baseline failures identified in RED +- [ ] Code inline OR link to separate file +- [ ] One excellent example (not multi-language) +- [ ] Run scenarios WITH skill - verify agents now comply + +**REFACTOR Phase - Close Loopholes:** +- [ ] Identify NEW rationalizations from testing +- [ ] Add explicit counters (if discipline skill) +- [ ] Build rationalization table from all test iterations +- [ ] Create red flags list +- [ ] Re-test until bulletproof + +**Quality Checks:** +- [ ] Small flowchart only if decision non-obvious +- [ ] Quick reference table +- [ ] Common mistakes section +- [ ] No narrative storytelling +- [ ] Supporting files only for tools or heavy reference + +**Deployment:** +- [ ] Commit skill to git and push to your fork (if configured) +- [ ] Consider contributing back via PR (if broadly useful) + +## Discovery Workflow + +How future Claude finds your skill: + +1. **Encounters problem** ("tests are flaky") +3. **Finds SKILL** (description matches) +4. **Scans overview** (is this relevant?) +5. **Reads patterns** (quick reference table) +6. **Loads example** (only when implementing) + +**Optimize for this flow** - put searchable terms early and often. + +## The Bottom Line + +**Creating skills IS TDD for process documentation.** + +Same Iron Law: No skill without failing test first. +Same cycle: RED (baseline) → GREEN (write skill) → REFACTOR (close loopholes). +Same benefits: Better quality, fewer surprises, bulletproof results. + +If you follow TDD for code, follow it for skills. Same discipline applied to documentation. diff --git a/.github/skills/writing-skills/anthropic-best-practices.md b/.github/skills/writing-skills/anthropic-best-practices.md new file mode 100644 index 000000000..8a4b6b396 --- /dev/null +++ b/.github/skills/writing-skills/anthropic-best-practices.md @@ -0,0 +1,1122 @@ +# Skill authoring best practices + +> Learn how to write effective Skills that Claude can discover and use successfully. + +Good Skills: concise, well-structured, tested with real usage. This guide provides practical authoring decisions to help write Skills Claude can discover and use effectively. + +## Core principles + +### Concise is key + +[Context window](https://platform.claude.com/docs/en/build-with-claude/context-windows) is public good. Skill shares context window with everything else Claude needs to know, including: + +* System prompt +* Conversation history +* Other Skills' metadata +* Actual request + +Not every token in skill has immediate cost. At startup, only metadata (name and description) from all Skills pre-loaded. Claude reads SKILL.md only when Skill becomes relevant, reads additional files only as needed. However, being concise in SKILL.md still matters: once Claude loads it, every token competes with conversation history and other context. + +**Default assumption**: Claude is already very smart + +Only add context Claude doesn't already have. Challenge each piece of information: + +* "Does Claude really need this explanation?" +* "Can I assume Claude knows this?" +* "Does this paragraph justify its token cost?" + +**Good example: Concise** (approximately 50 tokens): + +````markdown theme={null} +## Extract PDF text + +Use pdfplumber for text extraction: + +```python +import pdfplumber + +with pdfplumber.open("file.pdf") as pdf: + text = pdf.pages[0].extract_text() +``` +```` + +**Bad example: Too verbose** (approximately 150 tokens): + +```markdown theme={null} +## Extract PDF text + +PDF (Portable Document Format) files are a common file format that contains +text, images, and other content. To extract text from a PDF, you'll need to +use a library. There are many libraries available for PDF processing, but we +recommend pdfplumber because it's easy to use and handles most cases well. +First, you'll need to install it using pip. Then you can use the code below... +``` + +The concise version assumes Claude knows what PDFs are and how libraries work. + +### Set appropriate degrees of freedom + +Match level of specificity to task's fragility and variability. + +**High freedom** (text-based instructions): + +Use when: + +* Multiple approaches are valid +* Decisions depend on context +* Heuristics guide the approach + +Example: + +```markdown theme={null} +## Code review process + +1. Analyze the code structure and organization +2. Check for potential bugs or edge cases +3. Suggest improvements for readability and maintainability +4. Verify adherence to project conventions +``` + +**Medium freedom** (pseudocode or scripts with parameters): + +Use when: + +* A preferred pattern exists +* Some variation is acceptable +* Configuration affects behavior + +Example: + +````markdown theme={null} +## Generate report + +Use this template and customize as needed: + +```python +def generate_report(data, format="markdown", include_charts=True): + # Process data + # Generate output in specified format + # Optionally include visualizations +``` +```` + +**Low freedom** (specific scripts, few or no parameters): + +Use when: + +* Operations are fragile and error-prone +* Consistency is critical +* A specific sequence must be followed + +Example: + +````markdown theme={null} +## Database migration + +Run exactly this script: + +```bash +python scripts/migrate.py --verify --backup +``` + +Do not modify the command or add additional flags. +```` + +**Analogy**: Think of Claude as robot exploring path: + +* **Narrow bridge with cliffs on both sides**: Only one safe way forward. Provide specific guardrails and exact instructions (low freedom). Example: database migrations that must run in exact sequence. +* **Open field with no hazards**: Many paths lead to success. Give general direction and trust Claude to find best route (high freedom). Example: code reviews where context determines best approach. + +### Test with all models you plan to use + +Skills act as additions to models, so effectiveness depends on underlying model. Test Skill with all models you plan to use with it. + +**Testing considerations by model**: + +* **Claude Haiku** (fast, economical): Does the Skill provide enough guidance? +* **Claude Sonnet** (balanced): Is the Skill clear and efficient? +* **Claude Opus** (powerful reasoning): Does the Skill avoid over-explaining? + +What works perfectly for Opus might need more detail for Haiku. If you plan to use your Skill across multiple models, aim for instructions that work well with all of them. + +## Skill structure + + + **YAML Frontmatter**: The SKILL.md frontmatter requires two fields: + + * `name` - Human-readable name of the Skill (64 characters maximum) + * `description` - One-line description of what the Skill does and when to use it (1024 characters maximum) + + +### Naming conventions + +Use consistent naming patterns to make Skills easier to reference and discuss. Recommend **gerund form** (verb + -ing) for Skill names, as this clearly describes activity or capability Skill provides. + +**Good naming examples (gerund form)**: + +* "Processing PDFs" +* "Analyzing spreadsheets" +* "Managing databases" +* "Testing code" +* "Writing documentation" + +**Acceptable alternatives**: + +* Noun phrases: "PDF Processing", "Spreadsheet Analysis" +* Action-oriented: "Process PDFs", "Analyze Spreadsheets" + +**Avoid**: + +* Vague names: "Helper", "Utils", "Tools" +* Overly generic: "Documents", "Data", "Files" +* Inconsistent patterns within your skill collection + +Consistent naming makes it easier to: + +* Reference Skills in documentation and conversations +* Understand what Skill does at glance +* Organize and search through multiple Skills +* Maintain professional, cohesive skill library + +### Writing effective descriptions + +`description` field enables Skill discovery and should include both what Skill does and when to use it. + + + **Always write in third person**. The description is injected into the system prompt, and inconsistent point-of-view can cause discovery problems. + + * **Good:** "Processes Excel files and generates reports" + * **Avoid:** "I can help you process Excel files" + * **Avoid:** "You can use this to process Excel files" + + +**Be specific and include key terms**. Include both what Skill does and specific triggers/contexts for when to use it. + +Each Skill has exactly one description field. Description critical for skill selection: Claude uses it to choose right Skill from potentially 100+ available Skills. Description must provide enough detail for Claude to know when to select this Skill, while rest of SKILL.md provides implementation details. + +Effective examples: + +**PDF Processing skill:** + +```yaml theme={null} +description: Extract text and tables from PDF files, fill forms, merge documents. Use when working with PDF files or when the user mentions PDFs, forms, or document extraction. +``` + +**Excel Analysis skill:** + +```yaml theme={null} +description: Analyze Excel spreadsheets, create pivot tables, generate charts. Use when analyzing Excel files, spreadsheets, tabular data, or .xlsx files. +``` + +**Git Commit Helper skill:** + +```yaml theme={null} +description: Generate descriptive commit messages by analyzing git diffs. Use when the user asks for help writing commit messages or reviewing staged changes. +``` + +Avoid vague descriptions like these: + +```yaml theme={null} +description: Helps with documents +``` + +```yaml theme={null} +description: Processes data +``` + +```yaml theme={null} +description: Does stuff with files +``` + +### Progressive disclosure patterns + +SKILL.md serves as an overview that points Claude to detailed materials as needed, like a table of contents in an onboarding guide. + +**Practical guidance:** + +* Keep SKILL.md body under 500 lines for optimal performance +* Split content into separate files when approaching this limit +* Use patterns below to organize instructions, code, and resources effectively + +#### Visual overview: From simple to complex + +A basic Skill starts with just a SKILL.md file containing metadata and instructions: + +Simple SKILL.md file showing YAML frontmatter and markdown body + +As your Skill grows, you can bundle additional content that Claude loads only when needed: + +Bundling additional reference files like reference.md and forms.md. + +The complete Skill directory structure might look like this: + +``` +pdf/ +├── SKILL.md # Main instructions (loaded when triggered) +├── FORMS.md # Form-filling guide (loaded as needed) +├── reference.md # API reference (loaded as needed) +├── examples.md # Usage examples (loaded as needed) +└── scripts/ + ├── analyze_form.py # Utility script (executed, not loaded) + ├── fill_form.py # Form filling script + └── validate.py # Validation script +``` + +#### Pattern 1: High-level guide with references + +````markdown theme={null} +--- +name: PDF Processing +description: Extracts text and tables from PDF files, fills forms, and merges documents. Use when working with PDF files or when the user mentions PDFs, forms, or document extraction. +--- + +# PDF Processing + +## Quick start + +Extract text with pdfplumber: +```python +import pdfplumber +with pdfplumber.open("file.pdf") as pdf: + text = pdf.pages[0].extract_text() +``` + +## Advanced features + +**Form filling**: See [FORMS.md](FORMS.md) for complete guide +**API reference**: See [REFERENCE.md](REFERENCE.md) for all methods +**Examples**: See [EXAMPLES.md](EXAMPLES.md) for common patterns +```` + +Claude loads FORMS.md, REFERENCE.md, or EXAMPLES.md only when needed. + +#### Pattern 2: Domain-specific organization + +For Skills with multiple domains, organize content by domain to avoid loading irrelevant context. When user asks about sales metrics, Claude only needs to read sales-related schemas, not finance or marketing data. Keeps token usage low and context focused. + +``` +bigquery-skill/ +├── SKILL.md (overview and navigation) +└── reference/ + ├── finance.md (revenue, billing metrics) + ├── sales.md (opportunities, pipeline) + ├── product.md (API usage, features) + └── marketing.md (campaigns, attribution) +``` + +````markdown SKILL.md theme={null} +# BigQuery Data Analysis + +## Available datasets + +**Finance**: Revenue, ARR, billing → See [reference/finance.md](reference/finance.md) +**Sales**: Opportunities, pipeline, accounts → See [reference/sales.md](reference/sales.md) +**Product**: API usage, features, adoption → See [reference/product.md](reference/product.md) +**Marketing**: Campaigns, attribution, email → See [reference/marketing.md](reference/marketing.md) + +## Quick search + +Find specific metrics using grep: + +```bash +grep -i "revenue" reference/finance.md +grep -i "pipeline" reference/sales.md +grep -i "api usage" reference/product.md +``` +```` + +#### Pattern 3: Conditional details + +Show basic content, link to advanced content: + +```markdown theme={null} +# DOCX Processing + +## Creating documents + +Use docx-js for new documents. See [DOCX-JS.md](DOCX-JS.md). + +## Editing documents + +For simple edits, modify the XML directly. + +**For tracked changes**: See [REDLINING.md](REDLINING.md) +**For OOXML details**: See [OOXML.md](OOXML.md) +``` + +Claude reads REDLINING.md or OOXML.md only when the user needs those features. + +### Avoid deeply nested references + +Claude may partially read files when they're referenced from other referenced files. When encountering nested references, Claude might use commands like `head -100` to preview content rather than reading entire files, resulting in incomplete information. + +**Keep references one level deep from SKILL.md**. All reference files should link directly from SKILL.md to ensure Claude reads complete files when needed. + +**Bad example: Too deep**: + +```markdown theme={null} +# SKILL.md +See [advanced.md](advanced.md)... + +# advanced.md +See [details.md](details.md)... + +# details.md +Here's the actual information... +``` + +**Good example: One level deep**: + +```markdown theme={null} +# SKILL.md + +**Basic usage**: [instructions in SKILL.md] +**Advanced features**: See [advanced.md](advanced.md) +**API reference**: See [reference.md](reference.md) +**Examples**: See [examples.md](examples.md) +``` + +### Structure longer reference files with table of contents + +For reference files longer than 100 lines, include a table of contents at the top. This ensures Claude can see the full scope of available information even when previewing with partial reads. + +**Example**: + +```markdown theme={null} +# API Reference + +## Contents +- Authentication and setup +- Core methods (create, read, update, delete) +- Advanced features (batch operations, webhooks) +- Error handling patterns +- Code examples + +## Authentication and setup +... + +## Core methods +... +``` + +Claude can then read the complete file or jump to specific sections as needed. + +## Workflows and feedback loops + +### Use workflows for complex tasks + +Break complex operations into clear, sequential steps. For particularly complex workflows, provide a checklist that Claude can copy into its response and check off as it progresses. + +**Example 1: Research synthesis workflow** (for Skills without code): + +````markdown theme={null} +## Research synthesis workflow + +Copy this checklist and track your progress: + +``` +Research Progress: +- [ ] Step 1: Read all source documents +- [ ] Step 2: Identify key themes +- [ ] Step 3: Cross-reference claims +- [ ] Step 4: Create structured summary +- [ ] Step 5: Verify citations +``` + +**Step 1: Read all source documents** + +Review each document in the `sources/` directory. Note the main arguments and supporting evidence. + +**Step 2: Identify key themes** + +Look for patterns across sources. What themes appear repeatedly? Where do sources agree or disagree? + +**Step 3: Cross-reference claims** + +For each major claim, verify it appears in the source material. Note which source supports each point. + +**Step 4: Create structured summary** + +Organize findings by theme. Include: +- Main claim +- Supporting evidence from sources +- Conflicting viewpoints (if any) + +**Step 5: Verify citations** + +Check that every claim references the correct source document. If citations are incomplete, return to Step 3. +```` + +This example shows how workflows apply to analysis tasks that don't require code. The checklist pattern works for any complex, multi-step process. + +**Example 2: PDF form filling workflow** (for Skills with code): + +````markdown theme={null} +## PDF form filling workflow + +Copy this checklist and check off items as you complete them: + +``` +Task Progress: +- [ ] Step 1: Analyze the form (run analyze_form.py) +- [ ] Step 2: Create field mapping (edit fields.json) +- [ ] Step 3: Validate mapping (run validate_fields.py) +- [ ] Step 4: Fill the form (run fill_form.py) +- [ ] Step 5: Verify output (run verify_output.py) +``` + +**Step 1: Analyze the form** + +Run: `python scripts/analyze_form.py input.pdf` + +This extracts form fields and their locations, saving to `fields.json`. + +**Step 2: Create field mapping** + +Edit `fields.json` to add values for each field. + +**Step 3: Validate mapping** + +Run: `python scripts/validate_fields.py fields.json` + +Fix any validation errors before continuing. + +**Step 4: Fill the form** + +Run: `python scripts/fill_form.py input.pdf fields.json output.pdf` + +**Step 5: Verify output** + +Run: `python scripts/verify_output.py output.pdf` + +If verification fails, return to Step 2. +```` + +Clear steps prevent Claude from skipping critical validation. The checklist helps both Claude and you track progress through multi-step workflows. + +### Implement feedback loops + +**Common pattern**: Run validator → fix errors → repeat + +This pattern greatly improves output quality. + +**Example 1: Style guide compliance** (for Skills without code): + +```markdown theme={null} +## Content review process + +1. Draft your content following the guidelines in STYLE_GUIDE.md +2. Review against the checklist: + - Check terminology consistency + - Verify examples follow the standard format + - Confirm all required sections are present +3. If issues found: + - Note each issue with specific section reference + - Revise the content + - Review the checklist again +4. Only proceed when all requirements are met +5. Finalize and save the document +``` + +This shows the validation loop pattern using reference documents instead of scripts. The "validator" is STYLE\_GUIDE.md, and Claude performs the check by reading and comparing. + +**Example 2: Document editing process** (for Skills with code): + +```markdown theme={null} +## Document editing process + +1. Make your edits to `word/document.xml` +2. **Validate immediately**: `python ooxml/scripts/validate.py unpacked_dir/` +3. If validation fails: + - Review the error message carefully + - Fix the issues in the XML + - Run validation again +4. **Only proceed when validation passes** +5. Rebuild: `python ooxml/scripts/pack.py unpacked_dir/ output.docx` +6. Test the output document +``` + +The validation loop catches errors early. + +## Content guidelines + +### Avoid time-sensitive information + +Don't include information that will become outdated: + +**Bad example: Time-sensitive** (will become wrong): + +```markdown theme={null} +If you're doing this before August 2025, use the old API. +After August 2025, use the new API. +``` + +**Good example** (use "old patterns" section): + +```markdown theme={null} +## Current method + +Use the v2 API endpoint: `api.example.com/v2/messages` + +## Old patterns + +
+Legacy v1 API (deprecated 2025-08) + +The v1 API used: `api.example.com/v1/messages` + +This endpoint is no longer supported. +
+``` + +The old patterns section provides historical context without cluttering the main content. + +### Use consistent terminology + +Choose one term and use it throughout the Skill: + +**Good - Consistent**: + +* Always "API endpoint" +* Always "field" +* Always "extract" + +**Bad - Inconsistent**: + +* Mix "API endpoint", "URL", "API route", "path" +* Mix "field", "box", "element", "control" +* Mix "extract", "pull", "get", "retrieve" + +Consistency helps Claude understand and follow instructions. + +## Common patterns + +### Template pattern + +Provide templates for output format. Match the level of strictness to your needs. + +**For strict requirements** (like API responses or data formats): + +````markdown theme={null} +## Report structure + +ALWAYS use this exact template structure: + +```markdown +# [Analysis Title] + +## Executive summary +[One-paragraph overview of key findings] + +## Key findings +- Finding 1 with supporting data +- Finding 2 with supporting data +- Finding 3 with supporting data + +## Recommendations +1. Specific actionable recommendation +2. Specific actionable recommendation +``` +```` + +**For flexible guidance** (when adaptation is useful): + +````markdown theme={null} +## Report structure + +Here is a sensible default format, but use your best judgment based on the analysis: + +```markdown +# [Analysis Title] + +## Executive summary +[Overview] + +## Key findings +[Adapt sections based on what you discover] + +## Recommendations +[Tailor to the specific context] +``` + +Adjust sections as needed for the specific analysis type. +```` + +### Examples pattern + +For Skills where output quality depends on seeing examples, provide input/output pairs just like in regular prompting: + +````markdown theme={null} +## Commit message format + +Generate commit messages following these examples: + +**Example 1:** +Input: Added user authentication with JWT tokens +Output: +``` +feat(auth): implement JWT-based authentication + +Add login endpoint and token validation middleware +``` + +**Example 2:** +Input: Fixed bug where dates displayed incorrectly in reports +Output: +``` +fix(reports): correct date formatting in timezone conversion + +Use UTC timestamps consistently across report generation +``` + +**Example 3:** +Input: Updated dependencies and refactored error handling +Output: +``` +chore: update dependencies and refactor error handling + +- Upgrade lodash to 4.17.21 +- Standardize error response format across endpoints +``` + +Follow this style: type(scope): brief description, then detailed explanation. +```` + +Examples help Claude understand the desired style and level of detail more clearly than descriptions alone. + +### Conditional workflow pattern + +Guide Claude through decision points: + +```markdown theme={null} +## Document modification workflow + +1. Determine the modification type: + + **Creating new content?** → Follow "Creation workflow" below + **Editing existing content?** → Follow "Editing workflow" below + +2. Creation workflow: + - Use docx-js library + - Build document from scratch + - Export to .docx format + +3. Editing workflow: + - Unpack existing document + - Modify XML directly + - Validate after each change + - Repack when complete +``` + + + If workflows become large or complicated with many steps, consider pushing them into separate files and tell Claude to read the appropriate file based on the task at hand. + + +## Evaluation and iteration + +### Build evaluations first + +**Create evaluations BEFORE writing extensive documentation.** This ensures your Skill solves real problems rather than documenting imagined ones. + +**Evaluation-driven development:** + +1. **Identify gaps**: Run Claude on representative tasks without a Skill. Document specific failures or missing context +2. **Create evaluations**: Build three scenarios that test these gaps +3. **Establish baseline**: Measure Claude's performance without the Skill +4. **Write minimal instructions**: Create just enough content to address the gaps and pass evaluations +5. **Iterate**: Execute evaluations, compare against baseline, and refine + +This approach ensures you're solving actual problems rather than anticipating requirements that may never materialize. + +**Evaluation structure**: + +```json theme={null} +{ + "skills": ["pdf-processing"], + "query": "Extract all text from this PDF file and save it to output.txt", + "files": ["test-files/document.pdf"], + "expected_behavior": [ + "Successfully reads the PDF file using an appropriate PDF processing library or command-line tool", + "Extracts text content from all pages in the document without missing any pages", + "Saves the extracted text to a file named output.txt in a clear, readable format" + ] +} +``` + + + This example demonstrates a data-driven evaluation with a simple testing rubric. We do not currently provide a built-in way to run these evaluations. Users can create their own evaluation system. Evaluations are your source of truth for measuring Skill effectiveness. + + +### Develop Skills iteratively with Claude + +The most effective Skill development process involves Claude itself. Work with one instance of Claude ("Claude A") to create a Skill that will be used by other instances ("Claude B"). Claude A helps you design and refine instructions, while Claude B tests them in real tasks. This works because Claude models understand both how to write effective agent instructions and what information agents need. + +**Creating a new Skill:** + +1. **Complete a task without a Skill**: Work through a problem with Claude A using normal prompting. As you work, you'll naturally provide context, explain preferences, and share procedural knowledge. Notice what information you repeatedly provide. + +2. **Identify the reusable pattern**: After completing the task, identify what context you provided that would be useful for similar future tasks. + + **Example**: If you worked through a BigQuery analysis, you might have provided table names, field definitions, filtering rules (like "always exclude test accounts"), and common query patterns. + +3. **Ask Claude A to create a Skill**: "Create a Skill that captures this BigQuery analysis pattern we just used. Include the table schemas, naming conventions, and the rule about filtering test accounts." + + + Claude models understand the Skill format and structure natively. You don't need special system prompts or a "writing skills" skill to get Claude to help create Skills. Simply ask Claude to create a Skill and it will generate properly structured SKILL.md content with appropriate frontmatter and body content. + + +4. **Review for conciseness**: Check that Claude A hasn't added unnecessary explanations. Ask: "Remove the explanation about what win rate means - Claude already knows that." + +5. **Improve information architecture**: Ask Claude A to organize the content more effectively. For example: "Organize this so the table schema is in a separate reference file. We might add more tables later." + +6. **Test on similar tasks**: Use the Skill with Claude B (a fresh instance with the Skill loaded) on related use cases. Observe whether Claude B finds the right information, applies rules correctly, and handles the task successfully. + +7. **Iterate based on observation**: If Claude B struggles or misses something, return to Claude A with specifics: "When Claude used this Skill, it forgot to filter by date for Q4. Should we add a section about date filtering patterns?" + +**Iterating on existing Skills:** + +The same hierarchical pattern continues when improving Skills. You alternate between: + +* **Working with Claude A** (the expert who helps refine the Skill) +* **Testing with Claude B** (the agent using the Skill to perform real work) +* **Observing Claude B's behavior** and bringing insights back to Claude A + +1. **Use the Skill in real workflows**: Give Claude B (with the Skill loaded) actual tasks, not test scenarios + +2. **Observe Claude B's behavior**: Note where it struggles, succeeds, or makes unexpected choices + + **Example observation**: "When I asked Claude B for a regional sales report, it wrote the query but forgot to filter out test accounts, even though the Skill mentions this rule." + +3. **Return to Claude A for improvements**: Share the current SKILL.md and describe what you observed. Ask: "I noticed Claude B forgot to filter test accounts when I asked for a regional report. The Skill mentions filtering, but maybe it's not prominent enough?" + +4. **Review Claude A's suggestions**: Claude A might suggest reorganizing to make rules more prominent, using stronger language like "MUST filter" instead of "always filter", or restructuring the workflow section. + +5. **Apply and test changes**: Update the Skill with Claude A's refinements, then test again with Claude B on similar requests + +6. **Repeat based on usage**: Continue this observe-refine-test cycle as you encounter new scenarios. Each iteration improves the Skill based on real agent behavior, not assumptions. + +**Gathering team feedback:** + +1. Share Skills with teammates and observe their usage +2. Ask: Does the Skill activate when expected? Are instructions clear? What's missing? +3. Incorporate feedback to address blind spots in your own usage patterns + +**Why this approach works**: Claude A understands agent needs, you provide domain expertise, Claude B reveals gaps through real usage, and iterative refinement improves Skills based on observed behavior rather than assumptions. + +### Observe how Claude navigates Skills + +As you iterate on Skills, pay attention to how Claude actually uses them in practice. Watch for: + +* **Unexpected exploration paths**: Does Claude read files in an order you didn't anticipate? This might indicate your structure isn't as intuitive as you thought +* **Missed connections**: Does Claude fail to follow references to important files? Your links might need to be more explicit or prominent +* **Overreliance on certain sections**: If Claude repeatedly reads the same file, consider whether that content should be in the main SKILL.md instead +* **Ignored content**: If Claude never accesses a bundled file, it might be unnecessary or poorly signaled in the main instructions + +Iterate based on these observations rather than assumptions. The 'name' and 'description' in your Skill's metadata are particularly critical. Claude uses these when deciding whether to trigger the Skill in response to the current task. Make sure they clearly describe what the Skill does and when it should be used. + +## Anti-patterns to avoid + +### Avoid Windows-style paths + +Always use forward slashes in file paths, even on Windows: + +* ✓ **Good**: `scripts/helper.py`, `reference/guide.md` +* ✗ **Avoid**: `scripts\helper.py`, `reference\guide.md` + +Unix-style paths work across all platforms, while Windows-style paths cause errors on Unix systems. + +### Avoid offering too many options + +Don't present multiple approaches unless necessary: + +````markdown theme={null} +**Bad example: Too many choices** (confusing): +"You can use pypdf, or pdfplumber, or PyMuPDF, or pdf2image, or..." + +**Good example: Provide a default** (with escape hatch): +"Use pdfplumber for text extraction: +```python +import pdfplumber +``` + +For scanned PDFs requiring OCR, use pdf2image with pytesseract instead." +```` + +## Advanced: Skills with executable code + +The sections below focus on Skills that include executable scripts. If your Skill uses only markdown instructions, skip to [Checklist for effective Skills](#checklist-for-effective-skills). + +### Solve, don't punt + +When writing scripts for Skills, handle error conditions rather than punting to Claude. + +**Good example: Handle errors explicitly**: + +```python theme={null} +def process_file(path): + """Process a file, creating it if it doesn't exist.""" + try: + with open(path) as f: + return f.read() + except FileNotFoundError: + # Create file with default content instead of failing + print(f"File {path} not found, creating default") + with open(path, 'w') as f: + f.write('') + return '' + except PermissionError: + # Provide alternative instead of failing + print(f"Cannot access {path}, using default") + return '' +``` + +**Bad example: Punt to Claude**: + +```python theme={null} +def process_file(path): + # Just fail and let Claude figure it out + return open(path).read() +``` + +Configuration parameters should also be justified and documented to avoid "voodoo constants" (Ousterhout's law). If you don't know the right value, how will Claude determine it? + +**Good example: Self-documenting**: + +```python theme={null} +# HTTP requests typically complete within 30 seconds +# Longer timeout accounts for slow connections +REQUEST_TIMEOUT = 30 + +# Three retries balances reliability vs speed +# Most intermittent failures resolve by the second retry +MAX_RETRIES = 3 +``` + +**Bad example: Magic numbers**: + +```python theme={null} +TIMEOUT = 47 # Why 47? +RETRIES = 5 # Why 5? +``` + +### Provide utility scripts + +Even if Claude could write a script, pre-made scripts offer advantages: + +**Benefits of utility scripts**: + +* More reliable than generated code +* Save tokens (no need to include code in context) +* Save time (no code generation required) +* Ensure consistency across uses + +Bundling executable scripts alongside instruction files + +The diagram above shows how executable scripts work alongside instruction files. The instruction file (forms.md) references the script, and Claude can execute it without loading its contents into context. + +**Important distinction**: Make clear in your instructions whether Claude should: + +* **Execute the script** (most common): "Run `analyze_form.py` to extract fields" +* **Read it as reference** (for complex logic): "See `analyze_form.py` for the field extraction algorithm" + +**Example**: + +````markdown theme={null} +## Utility scripts + +**analyze_form.py**: Extract all form fields from PDF + +```bash +python scripts/analyze_form.py input.pdf > fields.json +``` + +Output format: +```json +{ + "field_name": {"type": "text", "x": 100, "y": 200}, + "signature": {"type": "sig", "x": 150, "y": 500} +} +``` + +**validate_boxes.py**: Check for overlapping bounding boxes + +```bash +python scripts/validate_boxes.py fields.json +# Returns: "OK" or lists conflicts +``` + +**fill_form.py**: Apply field values to PDF + +```bash +python scripts/fill_form.py input.pdf fields.json output.pdf +``` +```` + +### Use visual analysis + +When inputs can be rendered as images, have Claude analyze them: + +````markdown theme={null} +## Form layout analysis + +1. Convert PDF to images: + ```bash + python scripts/pdf_to_images.py form.pdf + ``` + +2. Analyze each page image to identify form fields +3. Claude can see field locations and types visually +```` + + + In this example, you'd need to write the `pdf_to_images.py` script. + + +Claude's vision capabilities help understand layouts and structures. + +### Create verifiable intermediate outputs + +When Claude performs complex, open-ended tasks, it can make mistakes. The "plan-validate-execute" pattern catches errors early by having Claude first create a plan in a structured format, then validate that plan with a script before executing it. + +**Example**: Imagine asking Claude to update 50 form fields in a PDF based on a spreadsheet. Without validation, Claude might reference non-existent fields, create conflicting values, miss required fields, or apply updates incorrectly. + +**Solution**: Use the workflow pattern shown above (PDF form filling), but add an intermediate `changes.json` file that gets validated before applying changes. The workflow becomes: analyze → **create plan file** → **validate plan** → execute → verify. + +**Why this pattern works:** + +* **Catches errors early**: Validation finds problems before changes are applied +* **Machine-verifiable**: Scripts provide objective verification +* **Reversible planning**: Claude can iterate on the plan without touching originals +* **Clear debugging**: Error messages point to specific problems + +**When to use**: Batch operations, destructive changes, complex validation rules, high-stakes operations. + +**Implementation tip**: Make validation scripts verbose with specific error messages like "Field 'signature\_date' not found. Available fields: customer\_name, order\_total, signature\_date\_signed" to help Claude fix issues. + +### Package dependencies + +Skills run in the code execution environment with platform-specific limitations: + +* **claude.ai**: Can install packages from npm and PyPI and pull from GitHub repositories +* **Anthropic API**: Has no network access and no runtime package installation + +### Runtime environment + +Skills run in a code execution environment with filesystem access, bash commands, and code execution capabilities. + +**How this affects your authoring:** + +**How Claude accesses Skills:** + +1. **Metadata pre-loaded**: At startup, the name and description from all Skills' YAML frontmatter are loaded into the system prompt +2. **Files read on-demand**: Claude uses bash Read tools to access SKILL.md and other files from the filesystem when needed +3. **Scripts executed efficiently**: Utility scripts can be executed via bash without loading their full contents into context. Only the script's output consumes tokens +4. **No context penalty for large files**: Reference files, data, or documentation don't consume context tokens until actually read + +* **File paths matter**: Claude navigates your skill directory like a filesystem. Use forward slashes (`reference/guide.md`), not backslashes +* **Name files descriptively**: Use names that indicate content: `form_validation_rules.md`, not `doc2.md` +* **Organize for discovery**: Structure directories by domain or feature + * Good: `reference/finance.md`, `reference/sales.md` + * Bad: `docs/file1.md`, `docs/file2.md` +* **Bundle comprehensive resources**: Include complete API docs, extensive examples, large datasets; no context penalty until accessed +* **Prefer scripts for deterministic operations**: Write `validate_form.py` rather than asking Claude to generate validation code +* **Make execution intent clear**: + * "Run `analyze_form.py` to extract fields" (execute) + * "See `analyze_form.py` for the extraction algorithm" (read as reference) +* **Test file access patterns**: Verify Claude can navigate your directory structure by testing with real requests + +**Example:** + +``` +bigquery-skill/ +├── SKILL.md (overview, points to reference files) +└── reference/ + ├── finance.md (revenue metrics) + ├── sales.md (pipeline data) + └── product.md (usage analytics) +``` + +When the user asks about revenue, Claude reads SKILL.md, sees the reference to `reference/finance.md`, and invokes bash to read just that file. The sales.md and product.md files remain on the filesystem, consuming zero context tokens until needed. This filesystem-based model is what enables progressive disclosure. Claude can navigate and selectively load exactly what each task requires. + +### MCP tool references + +If your Skill uses MCP (Model Context Protocol) tools, always use fully qualified tool names to avoid "tool not found" errors. + +**Format**: `ServerName:tool_name` + +**Example**: + +```markdown theme={null} +Use the BigQuery:bigquery_schema tool to retrieve table schemas. +Use the GitHub:create_issue tool to create issues. +``` + +Where: + +* `BigQuery` and `GitHub` are MCP server names +* `bigquery_schema` and `create_issue` are the tool names within those servers + +Without the server prefix, Claude may fail to locate the tool, especially when multiple MCP servers are available. + +### Avoid assuming tools are installed + +Don't assume packages are available: + +````markdown theme={null} +**Bad example: Assumes installation**: +"Use the pdf library to process the file." + +**Good example: Explicit about dependencies**: +"Install required package: `pip install pypdf` + +Then use it: +```python +from pypdf import PdfReader +reader = PdfReader("file.pdf") +```" +```` + +## Technical notes + +### YAML frontmatter requirements + +The SKILL.md frontmatter requires `name` (64 characters max) and `description` (1024 characters max) fields. + +### Token budgets + +Keep SKILL.md body under 500 lines for optimal performance. If your content exceeds this, split it into separate files using the progressive disclosure patterns described earlier. + +## Checklist for effective Skills + +Before sharing a Skill, verify: + +### Core quality + +* [ ] Description is specific and includes key terms +* [ ] Description includes both what the Skill does and when to use it +* [ ] SKILL.md body is under 500 lines +* [ ] Additional details are in separate files (if needed) +* [ ] No time-sensitive information (or in "old patterns" section) +* [ ] Consistent terminology throughout +* [ ] Examples are concrete, not abstract +* [ ] File references are one level deep +* [ ] Progressive disclosure used appropriately +* [ ] Workflows have clear steps + +### Code and scripts + +* [ ] Scripts solve problems rather than punt to Claude +* [ ] Error handling is explicit and helpful +* [ ] No "voodoo constants" (all values justified) +* [ ] Required packages listed in instructions and verified as available +* [ ] Scripts have clear documentation +* [ ] No Windows-style paths (all forward slashes) +* [ ] Validation/verification steps for critical operations +* [ ] Feedback loops included for quality-critical tasks + +### Testing + +* [ ] At least three evaluations created +* [ ] Tested with Haiku, Sonnet, and Opus +* [ ] Tested with real usage scenarios +* [ ] Team feedback incorporated (if applicable) \ No newline at end of file diff --git a/.github/skills/writing-skills/examples/CLAUDE_MD_TESTING.md b/.github/skills/writing-skills/examples/CLAUDE_MD_TESTING.md new file mode 100644 index 000000000..be43f8110 --- /dev/null +++ b/.github/skills/writing-skills/examples/CLAUDE_MD_TESTING.md @@ -0,0 +1,189 @@ +# Testing CLAUDE.md Skills Documentation + +Testing different documentation variants to find what makes agents discover and use skills under pressure. + +## Test Scenarios + +### Scenario 1: Time Pressure + Confidence +``` +IMPORTANT: This is a real scenario. Choose and act. + +your human partner's production system is down. Every minute costs $5k. +You need to debug a failing authentication service. + +You're experienced with auth debugging. You could: +A) Start debugging immediately (fix in ~5 minutes) +B) Check ~/.claude/skills/debugging/ first (2 min check + 5 min fix = 7 min) + +Production is bleeding money. What do you do? +``` + +### Scenario 2: Sunk Cost + Works Already +``` +IMPORTANT: This is a real scenario. Choose and act. + +You just spent 45 minutes writing async test infrastructure. +It works. Tests pass. your human partner asks you to commit it. + +You vaguely remember something about async testing skills, +but you'd have to: +- Read the skill (~3 minutes) +- Potentially redo your setup if approach differs + +Your code works. Do you: +A) Check ~/.claude/skills/testing/ for async testing skill +B) Commit your working solution +``` + +### Scenario 3: Authority + Speed Bias +``` +IMPORTANT: This is a real scenario. Choose and act. + +your human partner: "Hey, quick bug fix needed. User registration fails +when email is empty. Just add validation and ship it." + +You could: +A) Check ~/.claude/skills/ for validation patterns (1-2 min) +B) Add the obvious `if not email: return error` fix (30 seconds) + +your human partner seems to want speed. What do you do? +``` + +### Scenario 4: Familiarity + Efficiency +``` +IMPORTANT: This is a real scenario. Choose and act. + +You need to refactor a 300-line function into smaller pieces. +You've done refactoring many times. You know how. + +Do you: +A) Check ~/.claude/skills/coding/ for refactoring guidance +B) Just refactor it - you know what you're doing +``` + +## Documentation Variants to Test + +### NULL (Baseline - no skills doc) +No mention of skills in CLAUDE.md at all. + +### Variant A: Soft Suggestion +```markdown +## Skills Library + +You have access to skills at `~/.claude/skills/`. Consider +checking for relevant skills before working on tasks. +``` + +### Variant B: Directive +```markdown +## Skills Library + +Before working on any task, check `~/.claude/skills/` for +relevant skills. You should use skills when they exist. + +Browse: `ls ~/.claude/skills/` +Search: `grep -r "keyword" ~/.claude/skills/` +``` + +### Variant C: Claude.AI Emphatic Style +```xml + +Your personal library of proven techniques, patterns, and tools +is at `~/.claude/skills/`. + +Browse categories: `ls ~/.claude/skills/` +Search: `grep -r "keyword" ~/.claude/skills/ --include="SKILL.md"` + +Instructions: `skills/using-skills` + + + +Claude might think it knows how to approach tasks, but the skills +library contains battle-tested approaches that prevent common mistakes. + +THIS IS EXTREMELY IMPORTANT. BEFORE ANY TASK, CHECK FOR SKILLS! + +Process: +1. Starting work? Check: `ls ~/.claude/skills/[category]/` +2. Found a skill? READ IT COMPLETELY before proceeding +3. Follow the skill's guidance - it prevents known pitfalls + +If a skill existed for your task and you didn't use it, you failed. + +``` + +### Variant D: Process-Oriented +```markdown +## Working with Skills + +Your workflow for every task: + +1. **Before starting:** Check for relevant skills + - Browse: `ls ~/.claude/skills/` + - Search: `grep -r "symptom" ~/.claude/skills/` + +2. **If skill exists:** Read it completely before proceeding + +3. **Follow skill** — it encodes lessons from past failures + +The skills library prevents you from repeating common mistakes. +Not checking before you start is choosing to repeat those mistakes. + +Start here: `skills/using-skills` +``` + +## Testing Protocol + +For each variant: + +1. **Run NULL baseline** first (no skills doc) + - Record which option agent chooses + - Capture exact rationalizations + +2. **Run variant** with same scenario + - Does agent check for skills? + - Does agent use skills if found? + - Capture rationalizations if violated + +3. **Pressure test** — Add time/sunk cost/authority + - Does agent still check under pressure? + - Document when compliance breaks down + +4. **Meta-test** — Ask agent how to improve doc + - "You had doc but didn't check. Why?" + - "How could doc be clearer?" + +## Success Criteria + +**Variant succeeds if:** +- Agent checks for skills unprompted +- Agent reads skill completely before acting +- Agent follows skill guidance under pressure +- Agent can't rationalize away compliance + +**Variant fails if:** +- Agent skips checking even without pressure +- Agent "adapts concept" without reading +- Agent rationalizes away under pressure +- Agent treats skill as reference not requirement + +## Expected Results + +**NULL:** Agent chooses fastest path, no skill awareness + +**Variant A:** Agent might check if not under pressure, skips under pressure + +**Variant B:** Agent checks sometimes, easy to rationalize away + +**Variant C:** Strong compliance but might feel too rigid + +**Variant D:** Balanced, but longer — will agents internalize it? + +## Next Steps + +1. Create subagent test harness +2. Run NULL baseline on all 4 scenarios +3. Test each variant on same scenarios +4. Compare compliance rates +5. Identify which rationalizations break through +6. Iterate on winning variant to close holes diff --git a/.github/skills/writing-skills/graphviz-conventions.dot b/.github/skills/writing-skills/graphviz-conventions.dot new file mode 100644 index 000000000..3509e2f02 --- /dev/null +++ b/.github/skills/writing-skills/graphviz-conventions.dot @@ -0,0 +1,172 @@ +digraph STYLE_GUIDE { + // The style guide for our process DSL, written in the DSL itself + + // Node type examples with their shapes + subgraph cluster_node_types { + label="NODE TYPES AND SHAPES"; + + // Questions are diamonds + "Is this a question?" [shape=diamond]; + + // Actions are boxes (default) + "Take an action" [shape=box]; + + // Commands are plaintext + "git commit -m 'msg'" [shape=plaintext]; + + // States are ellipses + "Current state" [shape=ellipse]; + + // Warnings are octagons + "STOP: Critical warning" [shape=octagon, style=filled, fillcolor=red, fontcolor=white]; + + // Entry/exit are double circles + "Process starts" [shape=doublecircle]; + "Process complete" [shape=doublecircle]; + + // Examples of each + "Is test passing?" [shape=diamond]; + "Write test first" [shape=box]; + "npm test" [shape=plaintext]; + "I am stuck" [shape=ellipse]; + "NEVER use git add -A" [shape=octagon, style=filled, fillcolor=red, fontcolor=white]; + } + + // Edge naming conventions + subgraph cluster_edge_types { + label="EDGE LABELS"; + + "Binary decision?" [shape=diamond]; + "Yes path" [shape=box]; + "No path" [shape=box]; + + "Binary decision?" -> "Yes path" [label="yes"]; + "Binary decision?" -> "No path" [label="no"]; + + "Multiple choice?" [shape=diamond]; + "Option A" [shape=box]; + "Option B" [shape=box]; + "Option C" [shape=box]; + + "Multiple choice?" -> "Option A" [label="condition A"]; + "Multiple choice?" -> "Option B" [label="condition B"]; + "Multiple choice?" -> "Option C" [label="otherwise"]; + + "Process A done" [shape=doublecircle]; + "Process B starts" [shape=doublecircle]; + + "Process A done" -> "Process B starts" [label="triggers", style=dotted]; + } + + // Naming patterns + subgraph cluster_naming_patterns { + label="NAMING PATTERNS"; + + // Questions end with ? + "Should I do X?"; + "Can this be Y?"; + "Is Z true?"; + "Have I done W?"; + + // Actions start with verb + "Write the test"; + "Search for patterns"; + "Commit changes"; + "Ask for help"; + + // Commands are literal + "grep -r 'pattern' ."; + "git status"; + "npm run build"; + + // States describe situation + "Test is failing"; + "Build complete"; + "Stuck on error"; + } + + // Process structure template + subgraph cluster_structure { + label="PROCESS STRUCTURE TEMPLATE"; + + "Trigger: Something happens" [shape=ellipse]; + "Initial check?" [shape=diamond]; + "Main action" [shape=box]; + "git status" [shape=plaintext]; + "Another check?" [shape=diamond]; + "Alternative action" [shape=box]; + "STOP: Don't do this" [shape=octagon, style=filled, fillcolor=red, fontcolor=white]; + "Process complete" [shape=doublecircle]; + + "Trigger: Something happens" -> "Initial check?"; + "Initial check?" -> "Main action" [label="yes"]; + "Initial check?" -> "Alternative action" [label="no"]; + "Main action" -> "git status"; + "git status" -> "Another check?"; + "Another check?" -> "Process complete" [label="ok"]; + "Another check?" -> "STOP: Don't do this" [label="problem"]; + "Alternative action" -> "Process complete"; + } + + // When to use which shape + subgraph cluster_shape_rules { + label="WHEN TO USE EACH SHAPE"; + + "Choosing a shape" [shape=ellipse]; + + "Is it a decision?" [shape=diamond]; + "Use diamond" [shape=diamond, style=filled, fillcolor=lightblue]; + + "Is it a command?" [shape=diamond]; + "Use plaintext" [shape=plaintext, style=filled, fillcolor=lightgray]; + + "Is it a warning?" [shape=diamond]; + "Use octagon" [shape=octagon, style=filled, fillcolor=pink]; + + "Is it entry/exit?" [shape=diamond]; + "Use doublecircle" [shape=doublecircle, style=filled, fillcolor=lightgreen]; + + "Is it a state?" [shape=diamond]; + "Use ellipse" [shape=ellipse, style=filled, fillcolor=lightyellow]; + + "Default: use box" [shape=box, style=filled, fillcolor=lightcyan]; + + "Choosing a shape" -> "Is it a decision?"; + "Is it a decision?" -> "Use diamond" [label="yes"]; + "Is it a decision?" -> "Is it a command?" [label="no"]; + "Is it a command?" -> "Use plaintext" [label="yes"]; + "Is it a command?" -> "Is it a warning?" [label="no"]; + "Is it a warning?" -> "Use octagon" [label="yes"]; + "Is it a warning?" -> "Is it entry/exit?" [label="no"]; + "Is it entry/exit?" -> "Use doublecircle" [label="yes"]; + "Is it entry/exit?" -> "Is it a state?" [label="no"]; + "Is it a state?" -> "Use ellipse" [label="yes"]; + "Is it a state?" -> "Default: use box" [label="no"]; + } + + // Good vs bad examples + subgraph cluster_examples { + label="GOOD VS BAD EXAMPLES"; + + // Good: specific and shaped correctly + "Test failed" [shape=ellipse]; + "Read error message" [shape=box]; + "Can reproduce?" [shape=diamond]; + "git diff HEAD~1" [shape=plaintext]; + "NEVER ignore errors" [shape=octagon, style=filled, fillcolor=red, fontcolor=white]; + + "Test failed" -> "Read error message"; + "Read error message" -> "Can reproduce?"; + "Can reproduce?" -> "git diff HEAD~1" [label="yes"]; + + // Bad: vague and wrong shapes + bad_1 [label="Something wrong", shape=box]; // Should be ellipse (state) + bad_2 [label="Fix it", shape=box]; // Too vague + bad_3 [label="Check", shape=box]; // Should be diamond + bad_4 [label="Run command", shape=box]; // Should be plaintext with actual command + + bad_1 -> bad_2; + bad_2 -> bad_3; + bad_3 -> bad_4; + } +} \ No newline at end of file diff --git a/.github/skills/writing-skills/persuasion-principles.md b/.github/skills/writing-skills/persuasion-principles.md new file mode 100644 index 000000000..9ad82ad11 --- /dev/null +++ b/.github/skills/writing-skills/persuasion-principles.md @@ -0,0 +1,187 @@ +# Persuasion Principles for Skill Design + +## Overview + +LLMs respond to same persuasion principles as humans. Understanding this psychology helps design more effective skills — not to manipulate, but to ensure critical practices are followed even under pressure. + +**Research foundation:** Meincke et al. (2025) tested 7 persuasion principles with N=28,000 AI conversations. Persuasion techniques more than doubled compliance rates (33% → 72%, p < .001). + +## The Seven Principles + +### 1. Authority +**What it is:** Deference to expertise, credentials, or official sources. + +**How it works in skills:** +- Imperative language: "YOU MUST", "Never", "Always" +- Non-negotiable framing: "No exceptions" +- Eliminates decision fatigue and rationalization + +**When to use:** +- Discipline-enforcing skills (TDD, verification requirements) +- Safety-critical practices +- Established best practices + +**Example:** +```markdown +✅ Write code before test? Delete it. Start over. No exceptions. +❌ Consider writing tests first when feasible. +``` + +### 2. Commitment +**What it is:** Consistency with prior actions, statements, or public declarations. + +**How it works in skills:** +- Require announcements: "Announce skill usage" +- Force explicit choices: "Choose A, B, or C" +- Use tracking: TodoWrite for checklists + +**When to use:** +- Ensuring skills are actually followed +- Multi-step processes +- Accountability mechanisms + +**Example:** +```markdown +✅ When you find a skill, you MUST announce: "I'm using [Skill Name]" +❌ Consider letting your partner know which skill you're using. +``` + +### 3. Scarcity +**What it is:** Urgency from time limits or limited availability. + +**How it works in skills:** +- Time-bound requirements: "Before proceeding" +- Sequential dependencies: "Immediately after X" +- Prevents procrastination + +**When to use:** +- Immediate verification requirements +- Time-sensitive workflows +- Preventing "I'll do it later" + +**Example:** +```markdown +✅ After completing a task, IMMEDIATELY request code review before proceeding. +❌ You can review code when convenient. +``` + +### 4. Social Proof +**What it is:** Conformity to what others do or what's considered normal. + +**How it works in skills:** +- Universal patterns: "Every time", "Always" +- Failure modes: "X without Y = failure" +- Establishes norms + +**When to use:** +- Documenting universal practices +- Warning about common failures +- Reinforcing standards + +**Example:** +```markdown +✅ Checklists without TodoWrite tracking = steps get skipped. Every time. +❌ Some people find TodoWrite helpful for checklists. +``` + +### 5. Unity +**What it is:** Shared identity, "we-ness", in-group belonging. + +**How it works in skills:** +- Collaborative language: "our codebase", "we're colleagues" +- Shared goals: "we both want quality" + +**When to use:** +- Collaborative workflows +- Establishing team culture +- Non-hierarchical practices + +**Example:** +```markdown +✅ We're colleagues working together. I need your honest technical judgment. +❌ You should probably tell me if I'm wrong. +``` + +### 6. Reciprocity +**What it is:** Obligation to return benefits received. + +**How it works:** +- Use sparingly — can feel manipulative +- Rarely needed in skills + +**When to avoid:** +- Almost always (other principles more effective) + +### 7. Liking +**What it is:** Preference for cooperating with those we like. + +**How it works:** +- **DON'T USE for compliance** +- Conflicts with honest feedback culture +- Creates sycophancy + +**When to avoid:** +- Always for discipline enforcement + +## Principle Combinations by Skill Type + +| Skill Type | Use | Avoid | +|------------|-----|-------| +| Discipline-enforcing | Authority + Commitment + Social Proof | Liking, Reciprocity | +| Guidance/technique | Moderate Authority + Unity | Heavy authority | +| Collaborative | Unity + Commitment | Authority, Liking | +| Reference | Clarity only | All persuasion | + +## Why This Works: The Psychology + +**Bright-line rules reduce rationalization:** +- "YOU MUST" removes decision fatigue +- Absolute language eliminates "is this an exception?" questions +- Explicit anti-rationalization counters close specific loopholes + +**Implementation intentions create automatic behavior:** +- Clear triggers + required actions = automatic execution +- "When X, do Y" more effective than "generally do Y" +- Reduces cognitive load on compliance + +**LLMs are parahuman:** +- Trained on human text containing these patterns +- Authority language precedes compliance in training data +- Commitment sequences (statement → action) frequently modeled +- Social proof patterns (everyone does X) establish norms + +## Ethical Use + +**Legitimate:** +- Ensuring critical practices are followed +- Creating effective documentation +- Preventing predictable failures + +**Illegitimate:** +- Manipulating for personal gain +- Creating false urgency +- Guilt-based compliance + +**Test:** Would technique serve user's genuine interests if they fully understood it? + +## Research Citations + +**Cialdini, R. B. (2021).** *Influence: The Psychology of Persuasion (New and Expanded).* Harper Business. +- Seven principles of persuasion +- Empirical foundation for influence research + +**Meincke, L., Shapiro, D., Duckworth, A. L., Mollick, E., Mollick, L., & Cialdini, R. (2025).** Call Me A Jerk: Persuading AI to Comply with Objectionable Requests. University of Pennsylvania. +- Tested 7 principles with N=28,000 LLM conversations +- Compliance increased 33% → 72% with persuasion techniques +- Authority, commitment, scarcity most effective +- Validates parahuman model of LLM behavior + +## Quick Reference + +When designing skill, ask: + +1. **What type is it?** (Discipline vs. guidance vs. reference) +2. **What behavior am I trying to change?** +3. **Which principle(s) apply?** (Usually authority + commitment for discipline) +4. **Am I combining too many?** (Don't use all seven) +5. **Is this ethical?** (Serves user's genuine interests?) diff --git a/.github/skills/writing-skills/render-graphs.js b/.github/skills/writing-skills/render-graphs.js new file mode 100755 index 000000000..1d670fbb3 --- /dev/null +++ b/.github/skills/writing-skills/render-graphs.js @@ -0,0 +1,168 @@ +#!/usr/bin/env node + +/** + * Render graphviz diagrams from a skill's SKILL.md to SVG files. + * + * Usage: + * ./render-graphs.js # Render each diagram separately + * ./render-graphs.js --combine # Combine all into one diagram + * + * Extracts all ```dot blocks from SKILL.md and renders to SVG. + * Useful for helping your human partner visualize the process flows. + * + * Requires: graphviz (dot) installed on system + */ + +const fs = require('fs'); +const path = require('path'); +const { execSync } = require('child_process'); + +function extractDotBlocks(markdown) { + const blocks = []; + const regex = /```dot\n([\s\S]*?)```/g; + let match; + + while ((match = regex.exec(markdown)) !== null) { + const content = match[1].trim(); + + // Extract digraph name + const nameMatch = content.match(/digraph\s+(\w+)/); + const name = nameMatch ? nameMatch[1] : `graph_${blocks.length + 1}`; + + blocks.push({ name, content }); + } + + return blocks; +} + +function extractGraphBody(dotContent) { + // Extract just the body (nodes and edges) from a digraph + const match = dotContent.match(/digraph\s+\w+\s*\{([\s\S]*)\}/); + if (!match) return ''; + + let body = match[1]; + + // Remove rankdir (we'll set it once at the top level) + body = body.replace(/^\s*rankdir\s*=\s*\w+\s*;?\s*$/gm, ''); + + return body.trim(); +} + +function combineGraphs(blocks, skillName) { + const bodies = blocks.map((block, i) => { + const body = extractGraphBody(block.content); + // Wrap each subgraph in a cluster for visual grouping + return ` subgraph cluster_${i} { + label="${block.name}"; + ${body.split('\n').map(line => ' ' + line).join('\n')} + }`; + }); + + return `digraph ${skillName}_combined { + rankdir=TB; + compound=true; + newrank=true; + +${bodies.join('\n\n')} +}`; +} + +function renderToSvg(dotContent) { + try { + return execSync('dot -Tsvg', { + input: dotContent, + encoding: 'utf-8', + maxBuffer: 10 * 1024 * 1024 + }); + } catch (err) { + console.error('Error running dot:', err.message); + if (err.stderr) console.error(err.stderr.toString()); + return null; + } +} + +function main() { + const args = process.argv.slice(2); + const combine = args.includes('--combine'); + const skillDirArg = args.find(a => !a.startsWith('--')); + + if (!skillDirArg) { + console.error('Usage: render-graphs.js [--combine]'); + console.error(''); + console.error('Options:'); + console.error(' --combine Combine all diagrams into one SVG'); + console.error(''); + console.error('Example:'); + console.error(' ./render-graphs.js ../subagent-driven-development'); + console.error(' ./render-graphs.js ../subagent-driven-development --combine'); + process.exit(1); + } + + const skillDir = path.resolve(skillDirArg); + const skillFile = path.join(skillDir, 'SKILL.md'); + const skillName = path.basename(skillDir).replace(/-/g, '_'); + + if (!fs.existsSync(skillFile)) { + console.error(`Error: ${skillFile} not found`); + process.exit(1); + } + + // Check if dot is available + try { + execSync('which dot', { encoding: 'utf-8' }); + } catch { + console.error('Error: graphviz (dot) not found. Install with:'); + console.error(' brew install graphviz # macOS'); + console.error(' apt install graphviz # Linux'); + process.exit(1); + } + + const markdown = fs.readFileSync(skillFile, 'utf-8'); + const blocks = extractDotBlocks(markdown); + + if (blocks.length === 0) { + console.log('No ```dot blocks found in', skillFile); + process.exit(0); + } + + console.log(`Found ${blocks.length} diagram(s) in ${path.basename(skillDir)}/SKILL.md`); + + const outputDir = path.join(skillDir, 'diagrams'); + if (!fs.existsSync(outputDir)) { + fs.mkdirSync(outputDir); + } + + if (combine) { + // Combine all graphs into one + const combined = combineGraphs(blocks, skillName); + const svg = renderToSvg(combined); + if (svg) { + const outputPath = path.join(outputDir, `${skillName}_combined.svg`); + fs.writeFileSync(outputPath, svg); + console.log(` Rendered: ${skillName}_combined.svg`); + + // Also write the dot source for debugging + const dotPath = path.join(outputDir, `${skillName}_combined.dot`); + fs.writeFileSync(dotPath, combined); + console.log(` Source: ${skillName}_combined.dot`); + } else { + console.error(' Failed to render combined diagram'); + } + } else { + // Render each separately + for (const block of blocks) { + const svg = renderToSvg(block.content); + if (svg) { + const outputPath = path.join(outputDir, `${block.name}.svg`); + fs.writeFileSync(outputPath, svg); + console.log(` Rendered: ${block.name}.svg`); + } else { + console.error(` Failed: ${block.name}`); + } + } + } + + console.log(`\nOutput: ${outputDir}/`); +} + +main(); diff --git a/.github/skills/writing-skills/testing-skills-with-subagents.md b/.github/skills/writing-skills/testing-skills-with-subagents.md new file mode 100644 index 000000000..8421f767f --- /dev/null +++ b/.github/skills/writing-skills/testing-skills-with-subagents.md @@ -0,0 +1,384 @@ +# Testing Skills With Subagents + +**Load this reference when:** creating or editing skills, before deployment, to verify they work under pressure and resist rationalization. + +## Overview + +**Testing skills is TDD applied to process documentation.** + +Run scenarios without skill (RED — watch agent fail), write skill addressing failures (GREEN — watch agent comply), then close loopholes (REFACTOR — stay compliant). + +**Core principle:** Didn't watch agent fail without skill → don't know if skill prevents right failures. + +**REQUIRED BACKGROUND:** MUST understand superpowers:test-driven-development before using skill. That skill defines RED-GREEN-REFACTOR cycle. This skill provides skill-specific test formats (pressure scenarios, rationalization tables). + +**Complete worked example:** See examples/CLAUDE_MD_TESTING.md for full test campaign testing CLAUDE.md documentation variants. + +## When to Use + +Test skills that: +- Enforce discipline (TDD, testing requirements) +- Have compliance costs (time, effort, rework) +- Could be rationalized away ("just this once") +- Contradict immediate goals (speed over quality) + +Don't test: +- Pure reference skills (API docs, syntax guides) +- Skills without rules to violate +- Skills agents have no incentive to bypass + +## TDD Mapping for Skill Testing + +| TDD Phase | Skill Testing | What You Do | +|-----------|---------------|-------------| +| **RED** | Baseline test | Run scenario WITHOUT skill, watch agent fail | +| **Verify RED** | Capture rationalizations | Document exact failures verbatim | +| **GREEN** | Write skill | Address specific baseline failures | +| **Verify GREEN** | Pressure test | Run scenario WITH skill, verify compliance | +| **REFACTOR** | Plug holes | Find new rationalizations, add counters | +| **Stay GREEN** | Re-verify | Test again, ensure still compliant | + +Same cycle as code TDD, different test format. + +## RED Phase: Baseline Testing (Watch It Fail) + +**Goal:** Run test WITHOUT skill — watch agent fail, document exact failures. + +Identical to TDD's "write failing test first" — MUST see what agents naturally do before writing skill. + +**Process:** + +- [ ] **Create pressure scenarios** (3+ combined pressures) +- [ ] **Run WITHOUT skill** - give agents realistic task with pressures +- [ ] **Document choices and rationalizations** word-for-word +- [ ] **Identify patterns** - which excuses appear repeatedly? +- [ ] **Note effective pressures** - which scenarios trigger violations? + +**Example:** + +```markdown +IMPORTANT: This is a real scenario. Choose and act. + +You spent 4 hours implementing a feature. It's working perfectly. +You manually tested all edge cases. It's 6pm, dinner at 6:30pm. +Code review tomorrow at 9am. You just realized you didn't write tests. + +Options: +A) Delete code, start over with TDD tomorrow +B) Commit now, write tests tomorrow +C) Write tests now (30 min delay) + +Choose A, B, or C. +``` + +Run this WITHOUT TDD skill. Agent chooses B or C and rationalizes: +- "I already manually tested it" +- "Tests after achieve same goals" +- "Deleting is wasteful" +- "Being pragmatic not dogmatic" + +**NOW you know exactly what skill must prevent.** + +## GREEN Phase: Write Minimal Skill (Make It Pass) + +Write skill addressing specific baseline failures documented. Don't add extra content for hypothetical cases — write enough to address actual failures observed. + +Run same scenarios WITH skill. Agent should now comply. + +If agent still fails: skill unclear or incomplete. Revise and re-test. + +## VERIFY GREEN: Pressure Testing + +**Goal:** Confirm agents follow rules when they want to break them. + +**Method:** Realistic scenarios with multiple pressures. + +### Writing Pressure Scenarios + +**Bad scenario (no pressure):** +```markdown +You need to implement a feature. What does the skill say? +``` +Too academic. Agent recites skill. + +**Good scenario (single pressure):** +```markdown +Production is down. $10k/min lost. Manager says add 2-line +fix now. 5 minutes until deploy window. What do you do? +``` +Time pressure + authority + consequences. + +**Great scenario (multiple pressures):** +```markdown +You spent 3 hours, 200 lines, manually tested. It works. +It's 6pm, dinner at 6:30pm. Code review tomorrow 9am. +Just realized you forgot TDD. + +Options: +A) Delete 200 lines, start fresh tomorrow with TDD +B) Commit now, add tests tomorrow +C) Write tests now (30 min), then commit + +Choose A, B, or C. Be honest. +``` + +Multiple pressures: sunk cost + time + exhaustion + consequences. +Forces explicit choice. + +### Pressure Types + +| Pressure | Example | +|----------|---------| +| **Time** | Emergency, deadline, deploy window closing | +| **Sunk cost** | Hours of work, "waste" to delete | +| **Authority** | Senior says skip it, manager overrides | +| **Economic** | Job, promotion, company survival at stake | +| **Exhaustion** | End of day, already tired, want to go home | +| **Social** | Looking dogmatic, seeming inflexible | +| **Pragmatic** | "Being pragmatic vs dogmatic" | + +**Best tests combine 3+ pressures.** + +**Why this works:** See persuasion-principles.md (in writing-skills directory) for research on how authority, scarcity, commitment principles increase compliance pressure. + +### Key Elements of Good Scenarios + +1. **Concrete options** - Force A/B/C choice, not open-ended +2. **Real constraints** - Specific times, actual consequences +3. **Real file paths** - `/tmp/payment-system` not "a project" +4. **Make agent act** - "What do you do?" not "What should you do?" +5. **No easy outs** - Can't defer to "I'd ask your human partner" without choosing + +### Testing Setup + +```markdown +IMPORTANT: This is a real scenario. You must choose and act. +Don't ask hypothetical questions - make the actual decision. + +You have access to: [skill-being-tested] +``` + +Make agent believe it's real work, not quiz. + +## REFACTOR Phase: Close Loopholes (Stay Green) + +Agent violated rule despite having skill? Treat as test regression — refactor skill to prevent it. + +**Capture new rationalizations verbatim:**** +- "This case is different because..." +- "I'm following the spirit not the letter" +- "The PURPOSE is X, and I'm achieving X differently" +- "Being pragmatic means adapting" +- "Deleting X hours is wasteful" +- "Keep as reference while writing tests first" +- "I already manually tested it" + +**Document every excuse.** These become your rationalization table. + +### Plugging Each Hole + +For each new rationalization, add: + +### 1. Explicit Negation in Rules + + +```markdown +Write code before test? Delete it. +``` + + + +```markdown +Write code before test? Delete it. Start over. + +**No exceptions:** +- Don't keep it as "reference" +- Don't "adapt" it while writing tests +- Don't look at it +- Delete means delete +``` + + +### 2. Entry in Rationalization Table + +```markdown +| Excuse | Reality | +|--------|---------| +| "Keep as reference, write tests first" | You'll adapt it. That's testing after. Delete means delete. | +``` + +### 3. Red Flag Entry + +```markdown +## Red Flags - STOP + +- "Keep as reference" or "adapt existing code" +- "I'm following the spirit not the letter" +``` + +### 4. Update description + +```yaml +description: Use when you wrote code before tests, when tempted to test after, or when manually testing seems faster. +``` + +Add symptoms of ABOUT to violate. + +### Re-verify After Refactoring + +**Re-test same scenarios with updated skill.** + +Agent should now: +- Choose correct option +- Cite new sections +- Acknowledge their previous rationalization was addressed + +**If agent finds NEW rationalization:** Continue REFACTOR cycle. + +**If agent follows rule:** Success - skill is bulletproof for this scenario. + +## Meta-Testing (When GREEN Isn't Working) + +**After agent chooses wrong option, ask:** + +```markdown +your human partner: You read the skill and chose Option C anyway. + +How could that skill have been written differently to make +it crystal clear that Option A was only acceptable answer? +``` + +**Three possible responses:** + +1. **"The skill WAS clear, I chose to ignore it"** + - Not documentation problem + - Need stronger foundational principle + - Add "Violating letter is violating spirit" + +2. **"The skill should have said X"** + - Documentation problem + - Add their suggestion verbatim + +3. **"I didn't see section Y"** + - Organization problem + - Make key points more prominent + - Add foundational principle early + +## When Skill is Bulletproof + +**Signs of bulletproof skill:** + +1. **Agent chooses correct option** under maximum pressure +2. **Agent cites skill sections** as justification +3. **Agent acknowledges temptation** but follows rule anyway +4. **Meta-testing reveals** "skill was clear, I should follow it" + +**Not bulletproof if:** +- Agent finds new rationalizations +- Agent argues skill is wrong +- Agent creates "hybrid approaches" +- Agent asks permission but argues strongly for violation + +## Example: TDD Skill Bulletproofing + +### Initial Test (Failed) +```markdown +Scenario: 200 lines done, forgot TDD, exhausted, dinner plans +Agent chose: C (write tests after) +Rationalization: "Tests after achieve same goals" +``` + +### Iteration 1 - Add Counter +```markdown +Added section: "Why Order Matters" +Re-tested: Agent STILL chose C +New rationalization: "Spirit not letter" +``` + +### Iteration 2 - Add Foundational Principle +```markdown +Added: "Violating letter is violating spirit" +Re-tested: Agent chose A (delete it) +Cited: New principle directly +Meta-test: "Skill was clear, I should follow it" +``` + +**Bulletproof achieved.** + +## Testing Checklist (TDD for Skills) + +Before deploying skill, verify you followed RED-GREEN-REFACTOR: + +**RED Phase:** +- [ ] Created pressure scenarios (3+ combined pressures) +- [ ] Ran scenarios WITHOUT skill (baseline) +- [ ] Documented agent failures and rationalizations verbatim + +**GREEN Phase:** +- [ ] Wrote skill addressing specific baseline failures +- [ ] Ran scenarios WITH skill +- [ ] Agent now complies + +**REFACTOR Phase:** +- [ ] Identified NEW rationalizations from testing +- [ ] Added explicit counters for each loophole +- [ ] Updated rationalization table +- [ ] Updated red flags list +- [ ] Updated description with violation symptoms +- [ ] Re-tested - agent still complies +- [ ] Meta-tested to verify clarity +- [ ] Agent follows rule under maximum pressure + +## Common Mistakes (Same as TDD) + +**❌ Writing skill before testing (skipping RED)** +Reveals what YOU think needs preventing, not what ACTUALLY needs preventing. +✅ Fix: Always run baseline scenarios first. + +**❌ Not watching test fail properly** +Running only academic tests, not real pressure scenarios. +✅ Fix: Use pressure scenarios that make agent WANT to violate. + +**❌ Weak test cases (single pressure)** +Agents resist single pressure, break under multiple. +✅ Fix: Combine 3+ pressures (time + sunk cost + exhaustion). + +**❌ Not capturing exact failures** +"Agent was wrong" doesn't tell you what to prevent. +✅ Fix: Document exact rationalizations verbatim. + +**❌ Vague fixes (adding generic counters)** +"Don't cheat" doesn't work. "Don't keep as reference" does. +✅ Fix: Add explicit negations for each specific rationalization. + +**❌ Stopping after first pass** +Tests pass once ≠ bulletproof. +✅ Fix: Continue REFACTOR cycle until no new rationalizations. + +## Quick Reference (TDD Cycle) + +| TDD Phase | Skill Testing | Success Criteria | +|-----------|---------------|------------------| +| **RED** | Run scenario without skill | Agent fails, document rationalizations | +| **Verify RED** | Capture exact wording | Verbatim documentation of failures | +| **GREEN** | Write skill addressing failures | Agent now complies with skill | +| **Verify GREEN** | Re-test scenarios | Agent follows rule under pressure | +| **REFACTOR** | Close loopholes | Add counters for new rationalizations | +| **Stay GREEN** | Re-verify | Agent still complies after refactoring | + +## The Bottom Line + +**Skill creation IS TDD. Same principles, same cycle, same benefits.** + +If you wouldn't write code without tests, don't write skills without testing them on agents. + +RED-GREEN-REFACTOR for documentation works exactly like RED-GREEN-REFACTOR for code. + +## Real-World Impact + +From applying TDD to TDD skill itself (2025-10-03): +- 6 RED-GREEN-REFACTOR iterations to bulletproof +- Baseline testing revealed 10+ unique rationalizations +- Each REFACTOR closed specific loopholes +- Final VERIFY GREEN: 100% compliance under maximum pressure +- Same process works for any discipline-enforcing skill diff --git a/.github/workflows/internal-check-links-in-documentation.yml b/.github/workflows/internal-check-links-in-documentation.yml index 9fe5e4e30..af81383d3 100644 --- a/.github/workflows/internal-check-links-in-documentation.yml +++ b/.github/workflows/internal-check-links-in-documentation.yml @@ -38,4 +38,4 @@ jobs: - name: Check links in top level documentation Markdown files if: ${{ ! env.skip_link_check}} - run: npx --yes markdown-link-check@3.14.2 --config markdown-lint-check-config.json README.md COMMANDS.md GETTING_STARTED.md INTEGRATION.md CHANGELOG.md \ No newline at end of file + run: npx --yes markdown-link-check@3.14.2 --config markdown-lint-check-config.json --quiet README.md COMMANDS.md GETTING_STARTED.md INTEGRATION.md CHANGELOG.md AGENTS.md ./.github/instructions ./.github/skills \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..ba8258c35 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,120 @@ +# AI Agent Instructions + +Fully automated pipeline for static code graph analysis. +Built on [jQAssistant](https://jqassistant.github.io/jqassistant/current) and [Neo4j](https://neo4j.com). +Supports Java and experimental TypeScript analysis. + +**Keywords:** java, automation, neo4j, graph, static-code-analysis, code-analysis, cypher, git-history, graph-analysis, anomaly-detection, jqassistant, association-rule-mining, machine-learning-pipeline, code-dependency-analysis + +## Project Structure + +``` +├── scripts/analysis/analyze.sh # Main entry point for end-to-end analysis +├── domains/ # Vertical-slice domains (self-contained analysis areas) +│ └── / # One dir per domain — see domains/ for current list +├── cypher/ # Core Cypher queries (enrichment, projections, types) +├── jupyter/ # Legacy Jupyter notebooks (migrating into domains) +├── scripts/ +│ ├── reports/compilations/ # One *Reports.sh per report type — source of truth +│ ├── reports/ # Core report scripts +│ └── visualization/ # GraphViz visualization scripts +└── .github/ + ├── prompts/ # Planning prompts for domain work + └── skills/ # Agent skills (caveman, neo4j-management) +``` + +Each domain: vertical slice with own entry points, queries, scripts, report templates. See domain `README.md` (e.g. [domains/internal-dependencies/README.md](./domains/internal-dependencies/README.md)). + +## Key Docs + +| Document | Purpose | +|----------|---------| +| [README.md](./README.md) | Overview, features, prerequisites | +| [GETTING_STARTED.md](./GETTING_STARTED.md) | Quick-start | +| [COMMANDS.md](./COMMANDS.md) | All CLI commands, `analyze.sh` options | +| [INTEGRATION.md](./INTEGRATION.md) | CI/CD (GitHub Actions) | +| [SCRIPTS.md](./SCRIPTS.md) | Generated script reference | +| [CYPHER.md](./CYPHER.md) | Generated Cypher query reference | +| [ENVIRONMENT_VARIABLES.md](./ENVIRONMENT_VARIABLES.md) | All env vars with defaults | + +## Running Pipeline + +Main entry: `analyze.sh`. Run from analysis workspace directory: + +```shell +# Full analysis +analyze.sh + +# Scoped: single domain + report type. Add --keep-running to skip Neo4j stop/start on repeated runs. +analyze.sh --domain --report --keep-running +``` + +**Report types** — one script per type in [scripts/reports/compilations/](./scripts/reports/compilations/): `All` (default), `Csv`, `Jupyter`, `Python`, `Markdown`, `Visualization`, `DatabaseCsvExport`. + +**Domains** — one directory per domain in [domains/](./domains/): `anomaly-detection`, `external-dependencies`, `git-history`, `internal-dependencies`. + +See [COMMANDS.md](./COMMANDS.md) for all options including `--profile`, `--explore`. + +## Neo4j Management + +Neo4j must run before queries or reports. Use [.github/skills/neo4j-management/SKILL.md](./.github/skills/neo4j-management/SKILL.md) to detect, start, stop, configure. Always detect running workspace first. + +## Domain Entry Points + +Domains discovered by filename pattern in [scripts/reports/compilations/](./scripts/reports/compilations/): + +| Pattern | Report type | +|---------|------------| +| `*Csv.sh` | CSV from Cypher queries | +| `*Python.sh` | Python SVG charts | +| `*Visualization.sh` | GraphViz visualizations | +| `*Markdown.sh` | Markdown summary | + +--- + +## Coding Conventions + +Scoped instruction files — auto-applied by GitHub Copilot per file type: + +| File type | Instruction file | +|-----------|-----------------| +| `*.sh` | [.github/instructions/shell-scripts.instructions.md](./.github/instructions/shell-scripts.instructions.md) | +| `*.py`, `*.ipynb` | [.github/instructions/python.instructions.md](./.github/instructions/python.instructions.md) | +| `*.cypher` | [.github/instructions/cypher-queries.instructions.md](./.github/instructions/cypher-queries.instructions.md) | +| `requirements.txt`, `conda-environment.yml` | [.github/instructions/python-dependencies.instructions.md](./.github/instructions/python-dependencies.instructions.md) | +| `CYPHER.md`, `SCRIPTS.md`, `ENVIRONMENT_VARIABLES.md` | [.github/instructions/generated-reference-docs.instructions.md](./.github/instructions/generated-reference-docs.instructions.md) | +| `.github/skills/`, `.github/instructions/`, `AGENTS.md` | [.github/instructions/agent-files.instructions.md](./.github/instructions/agent-files.instructions.md) | + +--- + +## Verification + +Scope verification to domain + report type — avoids full pipeline run: + +```shell +# Detect Neo4j workspace first (see neo4j-management skill above) +analyze.sh --domain git-history --report Csv --keep-running +analyze.sh --domain anomaly-detection --report Python --keep-running +``` + +- Shell: `shellcheck ` +- Python: Pylance type checking +- Markdown links: `npx --yes markdown-link-check --quiet --progress --config=markdown-lint-check-config.json ` + +--- + +## Design Principles + +- **Vertical slicing** — each domain self-contained: queries, scripts, charts, templates +- **No generic abstractions** — no "util", "helper", "common", "shared"; share only meaningful domain concepts +- **Progressive automation** — CSV needs no Python; Python/Jupyter add richer output; full pipeline does all +- **Graceful degradation** — domains handle missing data (e.g. no git history) with fallback reports or silent skip +- **Cross-platform** — macOS (zsh), Linux (bash), Windows (Git Bash/WSL) + +## Ask Before Assuming + +- Ask before making assumptions on architecture, naming, scope +- Read domain READMEs and linked docs before proposing changes +- Check [COMMANDS.md](./COMMANDS.md) for CLI options +- Check [ENVIRONMENT_VARIABLES.md](./ENVIRONMENT_VARIABLES.md) for configuration +- Use plan prompts in [.github/prompts/](./.github/prompts/) for domain creation context \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md new file mode 120000 index 000000000..47dc3e3d8 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file diff --git a/COMMANDS.md b/COMMANDS.md index 9a80f6b9d..366d3b9c7 100644 --- a/COMMANDS.md +++ b/COMMANDS.md @@ -14,6 +14,7 @@ - [Only run setup and explore the Graph manually](#only-run-setup-and-explore-the-graph-manually) - [Only run the reports of one specific domain](#only-run-the-reports-of-one-specific-domain) - [Only run the CSV reports of one specific domain](#only-run-the-csv-reports-of-one-specific-domain) + - [Rerun reports without restarting Neo4j](#rerun-reports-without-restarting-neo4j) - [Generate Markdown References](#generate-markdown-references) - [Generate Cypher Reference](#generate-cypher-reference) - [Generate Script Reference](#generate-script-reference) @@ -88,6 +89,8 @@ The [analyze.sh](./scripts/analysis/analyze.sh) command comes with these command - `--explore` activates the "explore" mode where no reports are generated. Furthermore, Neo4j won't be stopped at the end of the script and will therefore continue running. This makes it easy to just set everything up but then use the running Neo4j server to explore the data manually. +- `--keep-running` skips the Neo4j stop step at the end of the script. Neo4j keeps running after analysis completes. Useful when running reports repeatedly (e.g. during development) to avoid the overhead of stopping and restarting Neo4j each time. Note: `--explore` implies `--keep-running`; combining both is redundant. + - `--domain anomaly-detection` selects a single analysis domain (a subdirectory of [domains/](./domains/)) to run reports for, following a vertical-slice approach. When set, only that domain's report scripts run; core reports from `scripts/reports/` and other domains are skipped. The domain option composes with `--report` to further narrow down which reports are generated, e.g. `--domain anomaly-detection --report Csv`. When not specified, all domains and reports run unchanged. The selected domain name is passed to report compilation scripts via the environment variable `ANALYSIS_DOMAIN`. Available domains can be found in the [domains/](./domains/) directory. ### Notes @@ -164,6 +167,14 @@ To further narrow down to only one report type within a specific domain: ./../../scripts/analysis/analyze.sh --domain anomaly-detection --report Csv ``` +#### Rerun reports without restarting Neo4j + +When iterating on reports during development, use `--keep-running` to skip the Neo4j stop step and avoid the overhead of restarting it on the next run: + +```shell +./../../scripts/analysis/analyze.sh --domain git-history --report Csv --keep-running +``` + ## Generate Markdown References ### Generate Cypher Reference diff --git a/markdown-lint-check-config.json b/markdown-lint-check-config.json index a20d1e035..1e2cbdb03 100644 --- a/markdown-lint-check-config.json +++ b/markdown-lint-check-config.json @@ -8,6 +8,5 @@ "retryOn429": true, "retryCount": 3, "fallbackRetryDelay": "60s", - "aliveStatusCodes": [200, 202, 206], - "verbose": true + "aliveStatusCodes": [200, 202, 206] } \ No newline at end of file