feat(compile): runtime prompt injection via prompt.js bundle

Default behaviour now renders the agent prompt at pipeline runtime via a
new `prompt.js` ado-script bundle, instead of embedding the body in the
compiled YAML. Body-only edits to the agent .md no longer require
recompiling the pipeline.

Set `inlined-imports: true` in front matter to opt out and keep the
legacy heredoc-embedded behaviour.

The runtime contract is a new `PromptSpec` IR (mirrors `GateSpec`):
schemars-derived JSON Schema, `json-schema-to-typescript` codegen into
`types-prompt.gen.ts`, base64-encoded into a single
`ADO_AW_PROMPT_SPEC` env var on the prompt.js step.

`prompt.js` reads the source from the workspace, strips front matter,
appends extension supplements, and substitutes `${{ parameters.NAME }}`
and `$(VAR)` patterns at runtime via env vars. Secret variables stay
secure-by-default (not auto-exposed).

Existing compiled YAMLs will fail `ado-aw check` after upgrade until
recompiled. Use `inlined-imports: true` as the one-line escape hatch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: jamesadevine

.github/workflows/ado-script.yml

@@ -39,9 +39,9 @@ jobs:
 
       - name: Verify generated TypeScript is up to date
         run: |
-          if ! git diff --exit-code -- scripts/ado-script/src/shared/types.gen.ts; then
+          if ! git diff --exit-code -- scripts/ado-script/src/shared/types.gen.ts scripts/ado-script/src/shared/types-prompt.gen.ts; then
             echo ""
-            echo "::error::types.gen.ts is out of date with the Rust IR."
+            echo "::error::Generated TS types are out of date with the Rust IR."
             echo "Run 'cd scripts/ado-script && npm run codegen' and commit the result."
             exit 1
           fi

.github/workflows/release.yml

@@ -69,8 +69,8 @@ jobs:
           npm ci
           npm run build
         # `npm run build` runs codegen + ncc + copies dist/gate/index.js
-        # to ../gate.js (i.e. scripts/gate.js), which is then included in
-        # scripts.zip by the next step.
+        # to ../gate.js and dist/prompt/index.js to ../prompt.js, both of
+        # which are then included in scripts.zip by the next step.
 
       - name: Package scripts bundle
         run: |

.gitignore

@@ -1,5 +1,6 @@
 target
 examples/sample-agent.yml
 scripts/gate.js
+scripts/prompt.js
 *.pyc
 __pycache__/

AGENTS.md

@@ -55,6 +55,7 @@ Every compiled pipeline runs as three sequential jobs:
 │   │   ├── gitattributes.rs # .gitattributes management for compiled pipelines
 │   │   ├── filter_ir.rs  # Filter expression IR: Fact/Predicate types, lowering, validation, codegen
 │   │   ├── pr_filters.rs # PR trigger filter generation (native ADO + gate steps)
+│   │   ├── prompt_ir.rs  # PromptSpec IR: schemars-typed runtime contract for prompt.js
 │   │   ├── extensions/   # CompilerExtension trait and infrastructure extensions
 │   │   │   ├── mod.rs    # Trait, Extension enum, collect_extensions(), re-exports
 │   │   │   ├── github.rs # Always-on GitHub MCP extension
@@ -120,8 +121,9 @@ Every compiled pipeline runs as three sequential jobs:
 ├── ado-aw-derive/        # Proc-macro crate: #[derive(SanitizeConfig)], #[derive(SanitizeContent)]
 ├── examples/             # Example agent definitions
 ├── scripts/              # Supporting scripts shipped as release artifacts
-│   ├── ado-script/       # TypeScript workspace for bundled gate.js (and future bundles)
-│   └── gate.js           # Bundled gate evaluator (built from scripts/ado-script/, see docs/ado-script.md)
+│   ├── ado-script/       # TypeScript workspace for bundled gate.js + prompt.js
+│   ├── gate.js           # Bundled gate evaluator (Setup job; see docs/ado-script.md)
+│   └── prompt.js         # Bundled prompt renderer (Agent job; see docs/ado-script.md)
 ├── tests/                # Integration tests and fixtures
 ├── docs/                 # Per-concept reference documentation (see index below)
 ├── Cargo.toml            # Rust dependencies
@@ -133,7 +135,7 @@ Every compiled pipeline runs as three sequential jobs:
 - **Language**: Rust (2024 edition) - Note: Rust 2024 edition exists and is the edition used by this project
 - **CLI Framework**: clap v4 with derive macros
 - **Error Handling**: anyhow for ergonomic error propagation
-- **Bundled scripts**: TypeScript + ncc (`scripts/ado-script/`) — compiled gate evaluator and future internal helpers; see [`docs/ado-script.md`](docs/ado-script.md).
+- **Bundled scripts**: TypeScript + ncc (`scripts/ado-script/`) — compiled gate evaluator (`gate.js`), prompt renderer (`prompt.js`), and future internal helpers; see [`docs/ado-script.md`](docs/ado-script.md).
 - **Async Runtime**: tokio with full features
 - **YAML Parsing**: serde_yaml
 - **MCP Server**: rmcp with server and transport-io features
@@ -185,8 +187,9 @@ index to jump to the right page.
   specification: `Fact`/`Predicate` types, three-pass compilation (lower →
   validate → codegen), gate step generation, adding new filter types.
 - [`docs/ado-script.md`](docs/ado-script.md) — `ado-script` workspace
-  (`scripts/ado-script/`): the bundled TypeScript runtime helpers (today:
-  `gate.js`), schemars-driven type codegen, and the A2 design decision.
+  (`scripts/ado-script/`): the bundled TypeScript runtime helpers
+  (`gate.js` for trigger gates, `prompt.js` for runtime prompt
+  rendering), schemars-driven type codegen, and the A2 design decision.
 - [`docs/local-development.md`](docs/local-development.md) — local development
   setup notes.
 

docs/ado-script.md

@@ -2,8 +2,13 @@
 
 `ado-script` is the umbrella name for **internal**, compiler-targeted
 TypeScript bundles that ado-aw emits into compiled pipelines as runtime
-helpers. The first (and currently only) bundle is **`gate.js`**, the
-trigger-filter gate evaluator.
+helpers. There are currently two bundles:
+
+- **`gate.js`** — the trigger-filter gate evaluator (Setup job).
+- **`prompt.js`** — the runtime prompt renderer that reads the agent
+  `.md` from the workspace, strips front matter, applies variable
+  substitution, and assembles the prompt with extension supplements
+  (Agent job; default behaviour as of v0.22).
 
 > Internal-only: `ado-script` is not a user-facing front-matter feature.
 > Authors do **not** write `ado-script:` blocks in their agent markdown.
@@ -46,26 +51,34 @@ scripts/ado-script/                # TS workspace
 ├── tsconfig.json                  # NodeNext, ESNext target
 ├── src/
 │   ├── shared/                    # Reusable across all bundles
-│   │   ├── types.gen.ts           # AUTO-GENERATED from Rust IR
+│   │   ├── types.gen.ts           # AUTO-GENERATED from Rust IR (gate)
+│   │   ├── types-prompt.gen.ts    # AUTO-GENERATED from Rust IR (prompt)
 │   │   ├── auth.ts                # ADO token / collection URI plumbing
 │   │   ├── ado-client.ts          # azure-devops-node-api wrapper + retries
 │   │   ├── env-facts.ts           # Pipeline-variable readers
 │   │   ├── policy.ts              # Failure-policy state machine
 │   │   └── vso-logger.ts          # ##vso[…] command emitters
-│   └── gate/                      # gate.js entry point
+│   ├── gate/                      # gate.js entry point
+│   │   ├── index.ts               # main()
+│   │   ├── bypass.ts              # build-reason auto-pass
+│   │   ├── facts.ts               # fact acquisition (env + REST)
+│   │   ├── predicates.ts          # 11 predicate evaluators
+│   │   └── selfcancel.ts          # best-effort build cancellation
+│   └── prompt/                    # prompt.js entry point
 │       ├── index.ts               # main()
-│       ├── bypass.ts              # build-reason auto-pass
-│       ├── facts.ts               # fact acquisition (env + REST)
-│       ├── predicates.ts          # 11 predicate evaluators
-│       └── selfcancel.ts          # best-effort build cancellation
-├── test/                          # End-to-end smoke tests
-└── dist/gate/index.js             # ncc-bundled output (gitignored)
+│       ├── frontmatter.ts         # YAML front-matter stripper
+│       └── substitute.ts          # ${{ parameters.* }} / $(VAR) substitution
+├── test/                          # End-to-end smoke tests (gate + prompt)
+└── dist/                          # ncc-bundled output (gitignored)
+    ├── gate/index.js
+    └── prompt/index.js
 ```
 
 The release workflow (`.github/workflows/release.yml`) runs `npm ci &&
-npm run build` and copies `dist/gate/index.js` to `scripts/gate.js`,
-which is then included in the `scripts.zip` release asset that pipelines
-download at runtime.
+npm run build` and copies `dist/gate/index.js` to `scripts/gate.js` and
+`dist/prompt/index.js` to `scripts/prompt.js`, both of which are then
+included in the `scripts.zip` release asset that pipelines download at
+runtime.
 
 ## Schema codegen — preventing drift
 
@@ -91,16 +104,20 @@ The pipeline:
                                         └──────────────────────────────┘
 ```
 
-`npm run codegen` runs both stages. The `ado-script` CI workflow
-(`.github/workflows/ado-script.yml`) regenerates the file and runs
+`npm run codegen` runs the full pipeline: it exports both the
+`gate-spec.schema.json` and `prompt-spec.schema.json` from Rust, then
+runs `json2ts` to regenerate `src/shared/types.gen.ts` (gate) and
+`src/shared/types-prompt.gen.ts` (prompt). The `ado-script` CI workflow
+(`.github/workflows/ado-script.yml`) regenerates both files and runs
 `git diff --exit-code` to fail on drift. If you change the IR shape in
 Rust, you must run `cd scripts/ado-script && npm run codegen` and
-commit the regenerated `types.gen.ts`.
+commit the regenerated `*.gen.ts` files.
 
-The Rust subcommand that emits the schema is intentionally hidden:
+The Rust subcommands that emit the schemas are intentionally hidden:
 
 ```sh
-cargo run -- export-gate-schema --output schema/gate-spec.schema.json
+cargo run -- export-gate-schema   --output schema/gate-spec.schema.json
+cargo run -- export-prompt-schema --output schema/prompt-spec.schema.json
 ```
 
 ## How the gate bundle is wired into emitted pipelines
@@ -120,6 +137,30 @@ steps when any `filters:` block is active:
 The IR-to-bash codegen lives in `compile_gate_step_external`
 (`src/compile/filter_ir.rs:~1100`).
 
+## How the prompt bundle is wired into emitted pipelines
+
+`generate_prepare_agent_prompt` in `src/compile/common.rs` injects a
+parallel three-step bundle into the **Agent job** when
+`inlined-imports: false` (the default):
+
+1. **`NodeTool@0`** — same Node 20.x install as for `gate.js`.
+   Idempotent across multiple invocations in the same job.
+2. **`curl` download** — fetches `scripts.zip` and extracts
+   `prompt.js` to `/tmp/ado-aw-scripts/prompt.js`. Each pool agent
+   downloads its own copy; the Setup and Agent jobs run on independent
+   agents so the download isn't shared.
+3. **`bash: node '/tmp/ado-aw-scripts/prompt.js'`** — runs the renderer
+   with `ADO_AW_PROMPT_SPEC` (base64 JSON of the `PromptSpec`) plus one
+   `ADO_AW_PARAM_<NAME>: ${{ parameters.<NAME> }}` env per declared
+   parameter.
+
+When `inlined-imports: true`, the same helper instead emits the legacy
+heredoc step that embeds the prompt body and supplements directly into
+the YAML; `prompt.js` is not invoked.
+
+Both download steps share the helper `scripts_download_step()` in
+`src/compile/extensions/mod.rs` so URL/version stay in lockstep.
+
 ## Adding a new internal use site
 
 Suppose we want a `poll.js` bundle (e.g. for polling external systems):
@@ -132,17 +173,28 @@ Suppose we want a `poll.js` bundle (e.g. for polling external systems):
    ```
    and extend `build` to also run it and copy `dist/poll/index.js` to
    `../poll.js`.
-3. Add tests under `src/poll/__tests__/`.
+3. Add tests under `src/poll/__tests__/` (unit) and `test/` (smoke,
+   gated on `dist/poll/index.js` existing).
 4. Wire from a new `CompilerExtension` (or extend an existing one) that
-   downloads and invokes `poll.js` as a runtime step.
-5. Update `.github/workflows/release.yml` if the zip exclusion list
+   downloads and invokes `poll.js` as a runtime step. Use the shared
+   `scripts_download_step()` and `node_tool_step()` helpers.
+5. If the contract is non-trivial, follow the `gate.js` /
+   `prompt.js` pattern: define a `Spec` struct in Rust with
+   `#[derive(Serialize, JsonSchema)]`, add a hidden
+   `export-poll-schema` CLI, and extend `npm run codegen` to regenerate
+   `types-poll.gen.ts`. For trivial contracts (a couple of env vars),
+   hand-written types are fine.
+6. Update `.github/workflows/release.yml` if the zip exclusion list
    needs to include the new `dist/poll` directory.
 
 ## Bundle-size budget
 
-Each bundled artifact must stay **under 5 MB**. The current `gate.js` is
-~1.1 MB, dominated by `azure-devops-node-api`. If a future bundle blows
-the budget:
+Each bundled artifact must stay **under 5 MB**. Current sizes:
+
+- `gate.js` is ~1.1 MB, dominated by `azure-devops-node-api`.
+- `prompt.js` is ~5 KB (no SDK dependency).
+
+If a future bundle blows the budget:
 
 - First, check ncc's `--minify` and `--target` flags.
 - If still too large, weigh dropping the SDK in favor of hand-rolled

docs/extending.md

@@ -53,6 +53,18 @@ Execution job (before the agent runs). `setup_steps()` injects into the Setup
 job (before the Execution job starts). Use `setup_steps()` for pre-activation
 gates or checks that must complete before the agent is launched.
 
+**`prompt_supplement()` and runtime substitution**: the markdown returned
+from this method is appended to the agent prompt at runtime by the
+`prompt.js` ado-script bundle. It is rendered through the same
+substitution pipeline as the user's prompt body, so supplements may
+contain `${{ parameters.NAME }}` (for declared parameters) and
+`$(VAR)` / `$(VAR.SUB)` references that will be resolved at pipeline
+runtime. See `docs/template-markers.md` for the full substitution
+contract. When `inlined-imports: true`, the supplement is instead
+embedded into the YAML at compile time via `wrap_prompt_append` and ADO
+substitution rules apply (no parameters, no runtime variable
+substitution).
+
 **Phase ordering**: Extensions are sorted by phase — runtimes
 (`ExtensionPhase::Runtime`) execute before tools (`ExtensionPhase::Tool`).
 This guarantees runtime install steps run before tool steps that may depend

docs/front-matter.md

@@ -135,6 +135,7 @@ parameters:                    # optional ADO runtime parameters (surfaced in UI
     displayName: "Clear agent memory"
     type: boolean
     default: false
+inlined-imports: false         # opt out of dynamic prompt injection (default: false)
 ---
 
 
@@ -143,6 +144,33 @@ parameters:                    # optional ADO runtime parameters (surfaced in UI
 Build the project and run all tests...
 ```
 
+## Dynamic Prompt Injection (`inlined-imports`)
+
+By default, the agent's prompt body is **not** embedded in the compiled
+pipeline YAML. Instead, the pipeline reads the source `.md` from the
+checked-out workspace at runtime, strips its front matter, applies
+variable substitution, and assembles the final prompt in a step backed
+by the `prompt.js` ado-script bundle. This means body-only edits to the
+agent's `.md` no longer require recompiling the pipeline.
+
+Set `inlined-imports: true` to opt out and embed the prompt body and
+extension supplements directly into the YAML at compile time (legacy
+behaviour). Use the inlined form when:
+
+- The Agent pool can't reach `github.com` to download `scripts.zip`.
+- You need a fully self-contained pipeline file (offline archival).
+- You want to inspect the exact rendered prompt by reading the YAML.
+
+Substitution patterns recognised at runtime by `prompt.js` (default
+mode only):
+
+| Pattern                       | Resolved via                                        | Notes                                                                                       |
+|-------------------------------|------------------------------------------------------|---------------------------------------------------------------------------------------------|
+| `${{ parameters.NAME }}`      | env `ADO_AW_PARAM_<NAME upper, hyphen→underscore>`   | Only declared parameters substitute; others left verbatim with a warning.                   |
+| `$(VAR)` / `$(VAR.SUB)`       | env `<name upper, dot→underscore>` (ADO native)      | Unset variables left verbatim with a warning. Secrets are not auto-exposed and stay verbatim. |
+| `$[ ... ]`                    | not substituted                                      | Left verbatim with one warning per render.                                                  |
+| `\$(...)`                     | escape                                               | Backslash stripped; `$(...)` left literal.                                                  |
+
 ## Workspace Defaults
 
 The `workspace:` field controls which directory the agent runs in. When it is

docs/template-markers.md

@@ -305,9 +305,29 @@ resources:
             - release/*
 ```
 
-## {{ agent_content }}
+## {{ prepare_agent_prompt }}
 
-Should be replaced with the markdown body (agent instructions) extracted from the source markdown file, excluding the YAML front matter. This content provides the agent with its task description and guidelines.
+Replaced with the YAML step(s) that prepare `/tmp/awf-tools/agent-prompt.md` for the agent.
+
+The expansion depends on the `inlined-imports` front-matter field:
+
+- **`inlined-imports: false`** (default) — emits a three-step bundle:
+  1. `NodeTool@0` to install Node 20.x.
+  2. A `curl` download of `scripts.zip` from the matching `githubnext/ado-aw` release, extracting `prompt.js` to `/tmp/ado-aw-scripts/prompt.js`.
+  3. A bash step that runs `node /tmp/ado-aw-scripts/prompt.js` with `ADO_AW_PROMPT_SPEC` (a base64-encoded `PromptSpec` JSON) and one `ADO_AW_PARAM_<NAME>: ${{ parameters.<NAME> }}` env entry per declared parameter. The renderer reads the source `.md` from the workspace, strips its front matter, applies variable substitution, appends extension supplements, and writes the result to `/tmp/awf-tools/agent-prompt.md`.
+
+  Substitution patterns recognised at runtime by `prompt.js`:
+
+  | Pattern                       | Resolved via                                        | Notes                                                                                       |
+  |-------------------------------|------------------------------------------------------|---------------------------------------------------------------------------------------------|
+  | `${{ parameters.NAME }}`      | env `ADO_AW_PARAM_<NAME upper, hyphen→underscore>`   | Only declared parameters substitute; others left verbatim with a warning.                   |
+  | `$(VAR)` / `$(VAR.SUB)`       | env `<name upper, dot→underscore>` (ADO native)      | Unset variables left verbatim with a warning. Secrets aren't auto-exposed and stay verbatim. |
+  | `$[ ... ]`                    | not substituted                                      | Left verbatim with one warning per render.                                                  |
+  | `\$(...)`                     | escape                                               | Backslash stripped; `$(...)` left literal.                                                  |
+
+- **`inlined-imports: true`** — emits a single legacy heredoc step that writes the markdown body verbatim into `/tmp/awf-tools/agent-prompt.md`. Extension prompt supplements are emitted as separate `cat >>` heredoc steps via `wrap_prompt_append`. No variable substitution beyond what ADO macros already do natively.
+
+This marker replaces the older `{{ agent_content }}` placeholder, which is no longer emitted by the compiler.
 
 ## {{ mcpg_config }}