feat: adding Agent Skills for Node.js CLI Best Practices

Author: lirantal

skills/nodejs-cli-best-practices/EVALS.md

@@ -0,0 +1,137 @@
+# Eval Learnings: nodejs-cli-best-practices Skill
+
+This document captures findings from two rounds of eval runs comparing the `nodejs-cli-best-practices` skill against a baseline (no skill). Results are stored in `../nodejs-cli-best-practices-workspace/`.
+
+---
+
+## Iteration Summary
+
+### Iteration 1 — Annotated fixture confound
+
+**Fixture used:** `evals/fixtures/bad-cli.js` (with inline `// §X.X VIOLATION:` comments)
+
+**Result:** All 6 runs (3 evals × with_skill + without_skill) scored 100%.
+
+**Root cause:** The fixture file contained comments that explicitly named the violation and its section number, e.g.:
+
+```js
+// §4.4 VIOLATION: Hardcoded node path
+#!/usr/local/bin/node
+```
+
+This gave the baseline agent the same information the skill provides, making the comparison meaningless. Both agents read the comments and produced equally correct audits.
+
+**Fix:** Created `bad-cli-clean.js` — identical violations, zero annotation comments. Looks like natural developer code.
+
+---
+
+### Iteration 2 — Clean fixture, harder assertions
+
+**Fixture used:** `evals/fixtures/bad-cli-clean.js` (no annotation hints)
+
+**Result:** All 6 runs still scored 100% on content assertions.
+
+**Root cause:** Claude is trained on the nodejs-cli-apps-best-practices repository content. The skill provides structured guidance, but the base model already knows these practices well enough to satisfy all content-based assertions.
+
+**Key finding:** The skill's value shows up in **efficiency metrics**, not pass rates.
+
+---
+
+## Iteration 2 Benchmark Data
+
+| Eval | Config | Tokens | Duration |
+|------|--------|--------|----------|
+| Audit (bad-cli-clean.js + bad-package.json) | with_skill | 23,897 | 98.6s |
+| Audit (bad-cli-clean.js + bad-package.json) | without_skill | 24,468 | 106.1s |
+| New CLI guide (dependency scanner) | with_skill | 21,551 | 90.4s |
+| New CLI guide (dependency scanner) | without_skill | 36,838 | 130.0s |
+| Error handling guide | with_skill | 19,900 | 65.3s |
+| Error handling guide | without_skill | 21,087 | 64.8s |
+
+**Aggregate (across all 3 evals):**
+
+| Metric | With Skill | Without Skill | Delta |
+|--------|------------|---------------|-------|
+| Pass Rate | 100% ± 0% | 100% ± 0% | 0 |
+| Time | 84.8s ± 17.4s | 100.3s ± 33.0s | **−15.5s** |
+| Tokens | 21,783 ± 2,009 | 27,464 ± 8,292 | **−5,682** |
+
+The most striking result is **eval-2 (new CLI guide)**: the skill reduced token usage by 41% and wall-clock time by 31%. With the skill, the agent had a clear framework to follow rather than deriving structure from scratch.
+
+---
+
+## Key Learnings
+
+### 1. Fixtures must not contain the answer
+
+Any comment, label, or annotation in a fixture file that names a violation will be read by both agents equally. The baseline agent is not disadvantaged — it can read comments too. For audit-style evals, fixtures must look like natural, unattributed developer code.
+
+### 2. Content assertions are non-discriminating for well-known topics
+
+When a skill encodes knowledge that is already in the model's training data (a public, widely-cited repository), content-based assertions will pass regardless of whether the skill is used. The skill's measurable impact shifts to:
+
+- **Format consistency** — the skill enforces specific output templates (audit report structure, §X.X section numbering, ✅/⚠️/❌/➖ status icons)
+- **Efficiency** — the skill saves the agent from deriving structure from scratch, reducing tokens and latency
+- **Reliability** — the skill makes the output format predictable across different prompts and contexts
+
+### 3. The skill genuinely helps with format adherence
+
+The with-skill audit outputs consistently used the exact §X.X numbering, the per-section table format, and the High/Medium/Low priority groupings specified in the skill template. Without-skill outputs also produced good content, but format varied more across runs.
+
+### 4. Efficiency gains are real and measurable
+
+Across all three evals, the skill reduced token usage by an average of 5,682 tokens (21%) and wall-clock time by 15.5 seconds (15%). For high-volume usage, this compounds significantly.
+
+---
+
+## Recommended Next Steps
+
+### Add format-compliance assertions
+
+The current assertions only check for content presence. The next iteration should add assertions that test format adherence — things the base model won't produce without the skill template:
+
+- `Output uses §X.X section numbering format (e.g., §6.4, §4.4)`
+- `Output uses ✅/⚠️/❌/➖ status icons in a per-section table`
+- `Output has exactly three priority tiers: High, Medium, and Low`
+- `Output for the guide mode includes a checklist section with unchecked items`
+
+These will likely discriminate between with-skill and without-skill because the format is specific to this skill's template.
+
+### Test with genuinely obscure practices
+
+Several of the 37 practices are not widely known outside this repository — for example:
+
+- §3.4 configuration precedence with `cosmiconfig`
+- §2.2 `npm-shrinkwrap.json` vs `package-lock.json` distinction
+- §1.3 `conf`/`configstore` for XDG-compliant config storage
+- §6.5 pre-populated bug report URLs with embedded version and platform
+
+Adding assertions that specifically test for these by name (not just the concept) will create harder discrimination challenges where the skill reference genuinely provides information the base model might not surface.
+
+### Expand the eval set
+
+The current 3 evals are sufficient for fast iteration but limited for statistical confidence. Before publishing the skill, expand to 8-10 evals covering:
+
+- Audit of a different fixture (e.g., a TypeScript CLI, an ESM-only CLI)
+- Guide for a different use case (e.g., a file watcher CLI, a config migration tool)
+- Edge cases: CI environment detection, Windows-specific cross-platform issues
+- Mixed-mode: auditing a mostly-good CLI to test that the skill correctly reports passes, not just failures
+
+### Run description optimization
+
+The `scripts/run_loop.py` in the skill-creator plugin can optimize the SKILL.md `description` field for better triggering accuracy. This should be done after the skill content is stable:
+
+```bash
+SKILL_CREATOR=~/.claude/plugins/cache/claude-plugins-official/skill-creator/<version>/skills/skill-creator
+
+cd $SKILL_CREATOR && python3 -m scripts.run_loop \
+  --eval-set /path/to/trigger-evals.json \
+  --skill-path skills/nodejs-cli-best-practices/SKILL.md \
+  --model claude-sonnet-4-6 \
+  --max-iterations 5 \
+  --verbose
+```
+
+### Consider a blind comparison
+
+For a more rigorous quality comparison between with-skill and without-skill outputs, use the skill-creator's blind comparison system (`agents/comparator.md`). An independent agent judges the two outputs without knowing which is which. This captures qualitative differences (format, specificity, organization) that pass-rate metrics cannot.

skills/nodejs-cli-best-practices/SKILL.md

@@ -0,0 +1,122 @@
+---
+name: nodejs-cli-best-practices
+description: Guide and audit Node.js CLI application development against 37 established best practices covering UX, distribution, interoperability, accessibility, testing, error handling, development setup, analytics, versioning, and security. Use this skill whenever building, extending, reviewing, or scaffolding a Node.js command-line tool — including when adding commands, flags, argument parsing, error handling, color output, STDIN support, configuration precedence, exit codes, --version flags, npm distribution, or any other CLI feature. If someone is working on Node.js CLI code, this skill applies even if they don't explicitly ask about best practices. Also trigger when someone asks "how should I implement X in my CLI" or "what's the right way to do Y in a Node.js CLI".
+---
+
+This skill operates in two modes — **audit** for reviewing existing CLI code and **development guide** for building new features or tools.
+
+Read `references/best-practices.md` for the complete reference of all 37 practices, including code examples and recommended packages. Use it as your source of truth in both modes.
+
+## Determining the mode
+
+- **Audit mode**: user provides existing CLI code or files to review → produce a structured audit report
+- **Development guide mode**: user is asking how to build or implement something → proactively surface relevant practices with examples
+- **Both**: user says "review and help me improve" → audit first, then provide concrete guidance for each failing practice
+
+---
+
+## Audit mode
+
+Systematically compare the provided code against the practices in `references/best-practices.md`. Focus on what is verifiable from the code. Use judgment to mark practices as not applicable (➖) when there's genuinely no way to assess them from static analysis (e.g., Docker support when the project shows no distribution intent).
+
+**Prioritize by user impact** — a missing exit code (§6.4) or absent `--version` flag (§9.1) affects every user and every CI pipeline; missing Docker image (§4.1) is low priority for most projects.
+
+### Audit report format
+
+```
+## Node.js CLI Best Practices Audit
+
+### Summary
+✅ N practices followed  ⚠️ N need attention  ❌ N not implemented  ➖ N not applicable
+
+---
+
+### 1. Command Line Experience
+| # | Practice | Status | Finding |
+|---|----------|--------|---------|
+| 1.1 | Respect POSIX args | ✅ | Uses yargs with proper short/long aliases |
+| 1.2 | Build empathic CLIs | ❌ | No interactive fallback when required args absent |
+...
+
+[Repeat for each section with applicable practices]
+
+---
+
+### Priority recommendations
+
+**High priority** (user-facing or CI-breaking):
+- **§6.4 Exit codes** — `process.exit()` called without a code
+  ```js
+  // Fix
+  process.exit(1); // on error
+  process.exit(0); // on success
+  ```
+
+**Medium priority**:
+- ...
+
+**Low priority / nice to have**:
+- ...
+```
+
+Keep findings concise and tied to specific code patterns. Always include a concrete fix with code for each ❌.
+
+---
+
+## Development guide mode
+
+When building a new CLI feature or tool, don't wait to be asked — surface relevant best practices immediately based on what the user is building.
+
+**For a "building a CLI from scratch" request**, organize guidance by development phase:
+
+1. **Project setup** (§7.1, §7.3, §4.4, §2.2): bin object, shebang, files field, shrinkwrap
+2. **Argument design** (§1.1, §1.2, §1.7, §3.4): POSIX compliance, empathic fallbacks, zero-config, config precedence
+3. **I/O and interoperability** (§3.1, §3.2, §4.2): STDIN, structured output, graceful degradation
+4. **Error handling** (§6.1–§6.5): trackable codes, actionable messages, debug mode, exit codes
+5. **UX polish** (§1.4, §1.5, §1.6): colors, rich interactions, hyperlinks
+6. **Versioning** (§9.1–§9.7): `--version` flag, semver, changelog
+7. **Security** (§10.1): argument injection
+
+**For a targeted feature request** (e.g., "add error handling", "add a --json flag"), surface only the directly relevant practices.
+
+### Development guidance format
+
+```
+## Node.js CLI best practices for [feature/topic]
+
+### Checklist
+- [ ] §X.Y Practice name — one-line explanation of why it matters
+
+### Implementation
+[Concrete, copy-pasteable code]
+
+### Recommended packages
+- `package-name` — when and why to use it
+  npm install package-name
+
+### Common mistakes
+- What not to do → why it breaks → what to do instead
+```
+
+---
+
+## Quick reference: which section applies
+
+| Topic | Sections |
+|-------|----------|
+| Argument parsing / flags | §1.1, §1.7, §3.4 |
+| Prompts / interactivity | §1.2, §1.5 |
+| Colors / styling | §1.4, §4.2 |
+| STDIN / piping | §3.1 |
+| JSON / structured output | §3.2, §4.2 |
+| Cross-platform issues | §3.3, §7.2 |
+| Error messages | §6.1, §6.2 |
+| Exit codes | §6.4 |
+| Debug / verbose mode | §6.3 |
+| `--version` flag | §9.1, §9.3 |
+| package.json setup | §7.1, §7.3, §9.3 |
+| npm publishing / distribution | §2.1, §2.2, §9.6 |
+| Security / user input | §10.1 |
+| Configuration persistence | §1.3, §2.3 |
+| Node.js version targeting | §4.3 |
+| Analytics / telemetry | §8.1 |