Skip to content

feat: make workflow capabilities self-documenting#100

Open
Whamp wants to merge 1 commit into
QuintinShaw:mainfrom
Whamp:feat/self-documenting-workflow-capabilities
Open

feat: make workflow capabilities self-documenting#100
Whamp wants to merge 1 commit into
QuintinShaw:mainfrom
Whamp:feat/self-documenting-workflow-capabilities

Conversation

@Whamp

@Whamp Whamp commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add one executable capability contract for workflow globals, tool inputs, option shapes, lifecycle constraints, static references, and dynamic catalogue ownership;
  • assemble project-owned workflow globals through that contract while preserving the existing runtime closures and VM behavior;
  • ship a package-discovered workflow-authoring skill with focused references and 11 runtime-tested examples;
  • generate capability publications and context measurements, then enforce version, package, link, coverage, runtime-alignment, and freshness checks during normal tests and publishing;
  • keep provider-backed comprehension optional while adding model-free replay, capability-coverage protection, and delivery-choice evidence.

Closes #65.

Implements the capability-contract design from Whamp/pi-dynamic-workflows#22. Adds evidence for the background-versus-inline decision discussed in #89 without changing #93's trigger authorization behavior.

Context impact

Compared with origin/main at 818fdd8:

Surface Main This PR Change
Permanent workflow prompt 766 bytes 742 bytes -24
Provider-visible workflow tool definition 9,558 bytes 3,802 bytes -5,756
Skill discovery 0 bytes 338 bytes +338
Ordinary workflow-owned always-on total 10,324 bytes 4,882 bytes -5,442 (-52.7%)

Detailed authoring guidance now loads on demand. In the comparative nine-model coverage sample, the optimized skill loaded a median 2,564 tokens versus 8,096 for the pre-trim skill, a 68.3% reduction.

See docs/workflow-prompt-guidance-rationale.md for the reviewed record of prompt insertions, removals, and compactions. See docs/workflow-authoring-evidence.md for methodology, model matrix, per-scenario results, and reproduction commands.

Capability and quality evidence

  • The capability contract inventories the 18 curated project-owned globals and all current workflow-tool inputs, including resumeFromRunId.
  • Runtime assembly rejects missing declared implementations, reports and ignores undeclared ones, and preserves implementation identity.
  • Live model and agentType values remain outside static prompts and generated references.
  • The installed skill contains focused write/edit/review/debug routes, exact generated facts, six orchestration patterns, four focused recipes, and a dedicated validated-gate() recipe.
  • Every stable authoring surface is either covered behaviorally or protected as guidance-frozen.
  • Optional comprehension prompts do not name the skill; generated workflows run through the real parser/runtime with deterministic fake agents.

Post-tuning provider-backed validation:

  • pre-trim skill: 73/81;
  • optimized skill: 74/81;
  • generate-and-filter: 27/27 optimized versus 26/27 pre-trim;
  • all seven configurations other than Qwen and Spark: 63/63 optimized;
  • protected GPT-5.4 Mini/Luna coverage gate: 18/18;
  • protected GPT-5.4 Mini/Luna six-scenario core gate: 36/36.

This is not an independent holdout: the same scenario families informed the guidance changes. An earlier corrected-harness sample scored pre-trim at 75/81 and optimized at 71/81, so provider scores remain sampled, non-blocking evidence. The final candidate did not repeat the earlier nine-model, 162-case core panel.

The delivery-choice harness has deterministic tests and an optional provider CLI; this PR reports no provider-run delivery-choice result. Provider calls are not part of normal tests, release checks, or publishing.

Compatibility

  • preserves feat: authorize keyword triggering instead of forcing it (addresses #88, #89) #93's single opt-in promptGuidelines gate and default background delivery;
  • preserves the existing truthy-object gate().ok compatibility path;
  • preserves the existing regex determinism blocklist and VM substrate;
  • keeps console available for saved workflows while teaching log();
  • changes no live model-route or agentType configuration format;
  • adds no migration state, feature flag, or dual-running path.

Verification

  • npm run check
  • npm run build
  • npm test1,027/1,027 passed
  • npm run docs:check
  • npm run context:check
  • npm run guidance:check
  • npm run release:verify0 warnings
  • npm pack --dry-run — 166 files; all 27 workflow-authoring skill files included
  • git diff --check
  • CodeGraph changed-file cycle and boundary checks

The PR is one atomic commit based on current origin/main. Raw .auto experiment data is intentionally excluded from both the commit and package.

@Whamp
Whamp force-pushed the feat/self-documenting-workflow-capabilities branch from 3248d99 to 4c137b7 Compare July 18, 2026 12:27

@QuintinShaw QuintinShaw left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seriously impressive piece of work, @Whamp. Reviewed and tested it end to end — the runtime capability contract, #93 preservation, the skill + examples, and the new release gates, plus a real-pi smoke on 0.80.10 — and it's in excellent shape: #93's arm/authorize path is untouched, the tool schema stays a single top-level object (no repeat of the recent provider-400), the runtime globals are behavior-preserved, 1027 tests and release:verify are green, and the roughly 9.5KB → 3.8KB context win is real and measured. The evidence discipline — generated docs checked for drift, examples that actually run, provider calls kept out of CI — is exactly right. Thank you for the depth here.

One small fast-follow, non-blocking: there's no refresh path for the frozen-guidance hashes — editing SKILL.md (or a frozen reference/example) trips a hard PROTECTED_GUIDANCE_DRIFT with no *:generate to recompute the hardcoded SHA-256s, and the error doesn't hint how to fix it. Is that manual step deliberate — a conscious "you're touching protected guidance" gate, in which case a line in CONTRIBUTING would cover it — or would you want a small guidance:freeze script to match your other *:generate surfaces? Happy to send either your way.

Otherwise this is good to go — give me the nod and I'll merge.

@Whamp

Whamp commented Jul 19, 2026

Copy link
Copy Markdown
Contributor Author

Ok thank you and great feedback let me see what I can do to clean that up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reduce always-on workflow prompt size (model catalogue adds ~5k tokens)

2 participants