feat(compile): runtime prompt loading via {{#runtime-import}} markers

Adopts gh-aw's {{#runtime-import path}} marker model and the inlined-imports front-matter toggle. Agent prompt bodies (and the Stage-2 threat-analysis prompt) are now loaded at pipeline runtime by default; body edits no longer require ado-aw compile.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: jamesadevine

.github/workflows/ado-script.yml

@@ -69,13 +69,13 @@ jobs:
         working-directory: scripts/ado-script
         run: npm run typecheck
 
-      - name: Build bundle (gate.js)
+      - name: Build bundles
         working-directory: scripts/ado-script
         run: npm run build
 
-      - name: Smoke-test bundle
+      - name: Smoke-test bundles
         working-directory: scripts/ado-script
-        run: npx vitest run -c vitest.config.smoke.ts
+        run: npm run test:smoke
 
       - name: E2E gate test
         run: cargo test --test gate_e2e -- --ignored --nocapture

AGENTS.md

@@ -61,7 +61,7 @@ Every compiled pipeline runs as three sequential jobs:
 │   │   │   ├── mod.rs    # Trait, Extension enum, collect_extensions(), re-exports
 │   │   │   ├── github.rs # Always-on GitHub MCP extension
 │   │   │   ├── safe_outputs.rs # Always-on SafeOutputs MCP extension
-│   │   │   ├── trigger_filters.rs # Trigger filter extension (gate evaluator delivery)
+│   │   │   ├── ado_script.rs # Always-on ado-script extension (gate evaluator + runtime-import resolver, per-job downloads)
 │   │   │   └── tests.rs  # Extension integration tests
 │   │   ├── codemods/     # Front-matter codemods (one file per transformation)
 │   │   │   ├── mod.rs    # Codemod struct, CODEMODS registry, runner
@@ -156,7 +156,9 @@ Every compiled pipeline runs as three sequential jobs:
 │   ├── update-ado-agentic-workflow.md # Guide for modifying an existing agentic pipeline
 │   └── debug-ado-agentic-workflow.md  # Guide for troubleshooting a failing agentic pipeline
 ├── scripts/              # Supporting scripts shipped as release artifacts
-│   ├── ado-script/       # TypeScript workspace for bundled gate.js (and future bundles)
+│   ├── ado-script/       # TypeScript workspace for bundled gate.js, import.js, and future bundles
+│   │   └── src/
+│   │       └── import/   # Runtime prompt resolver bundle
 │   └── gate.js           # Bundled gate evaluator (built from scripts/ado-script/, see docs/ado-script.md)
 ├── tests/                # Integration tests and fixtures
 ├── docs/                 # Per-concept reference documentation (see index below)
@@ -169,7 +171,7 @@ Every compiled pipeline runs as three sequential jobs:
 - **Language**: Rust (2024 edition) - Note: Rust 2024 edition exists and is the edition used by this project
 - **CLI Framework**: clap v4 with derive macros
 - **Error Handling**: anyhow for ergonomic error propagation
-- **Bundled scripts**: TypeScript + ncc (`scripts/ado-script/`) — compiled gate evaluator and future internal helpers; see [`docs/ado-script.md`](docs/ado-script.md).
+- **Bundled scripts**: TypeScript + ncc (`scripts/ado-script/`) — compiled gate evaluator, runtime import resolver, and future internal helpers; see [`docs/ado-script.md`](docs/ado-script.md).
 - **Async Runtime**: tokio with full features
 - **YAML Parsing**: serde_yaml
 - **MCP Server**: rmcp with server and transport-io features
@@ -193,6 +195,8 @@ index to jump to the right page.
 
 - [`docs/front-matter.md`](docs/front-matter.md) — full agent file format
   (markdown body + YAML front matter grammar) with every supported field.
+- [`docs/runtime-imports.md`](docs/runtime-imports.md) — runtime prompt import
+  markers, path resolution, and `inlined-imports:` behavior.
 - [`docs/schedule-syntax.md`](docs/schedule-syntax.md) — fuzzy schedule time
   syntax (`daily around 14:00`, `weekly on monday`, timezones, scattering).
 - [`docs/engine.md`](docs/engine.md) — `engine:` configuration (model,
@@ -239,7 +243,7 @@ index to jump to the right page.
   adding codemods.
 - [`docs/ado-script.md`](docs/ado-script.md) — `ado-script` workspace
   (`scripts/ado-script/`): the bundled TypeScript runtime helpers (today:
-  `gate.js`), schemars-driven type codegen, and the A2 design decision.
+  `gate.js` and `import.js`), schemars-driven type codegen, and the A2 design decision.
 - [`docs/local-development.md`](docs/local-development.md) — local development
   setup notes.
 

docs/ado-script.md

@@ -3,8 +3,9 @@
 `ado-script` is the umbrella name for the TypeScript workspace at
 [`scripts/ado-script/`](../scripts/ado-script/). It produces small,
 ncc-bundled Node programs that the **compiler injects into every emitted
-pipeline** as runtime helpers. The first (and currently only) bundle is
-`gate.js`, the trigger-filter gate evaluator.
+pipeline** as runtime helpers. Today it produces `gate.js`, the
+trigger-filter gate evaluator, and `import.js`, the runtime prompt
+resolver described in [`runtime-imports.md`](runtime-imports.md).
 
 > **Internal-only.** `ado-script` is not a user-facing front-matter
 > feature. Authors never write an `ado-script:` block in their agent
@@ -33,6 +34,33 @@ discriminated union. There is no `eval`, no `Function`, no `vm` — a
 compromised compiler cannot use the spec to run arbitrary code on the
 pipeline runner.
 
+## What `import.js` does
+
+`import.js` is a single-shot Node program. It reads the prompt file path
+from `argv[2]` and resolves `{{#runtime-import path}}` markers in place.
+The compiler runs it as a post-prepare-prompt step when
+[`inlined-imports: false`](front-matter.md#inlined-imports). See
+[`runtime-imports.md`](runtime-imports.md) for the author-facing marker
+syntax.
+
+### Env-var contract
+
+| Env var | Required | Purpose |
+|---|---|---|
+| `ADO_AW_IMPORT_BASE` | No | Base directory for relative imports. Defaults to `dirname(argv[2])`. |
+
+The bundle lives at `dist/import/index.js` and ships in the same
+`ado-script.zip` release asset as `gate.js`, so pipelines download it
+through the same Setup-job asset flow. `import.js` uses only the Node
+standard library, so the ncc bundle is small (~1.5 KB) and carries no
+SDK dependency.
+
+The Stage-2 threat-analysis prompt is **not** runtime-imported.
+`src/data/threat-analysis.md` is `include_str!`'d into the `ado-aw`
+binary and inlined into the emitted YAML at compile time, matching
+gh-aw's pattern (their `threat_detection.md` ships with the setup
+action and is read directly from disk — no marker, no resolver).
+
 ## End-to-end data flow
 
 ```
@@ -147,14 +175,19 @@ scripts/ado-script/
 │   │   ├── env-facts.ts         # Pipeline-variable readers + ENV_BY_FACT + BRANCH_FACTS + ref-prefix stripping
 │   │   ├── policy.ts            # PolicyTracker state machine
 │   │   └── vso-logger.ts        # ##vso[…] emitters with property/message escaping; complete() is idempotent
-│   └── gate/                    # gate.js entry point + per-concern modules
-│       ├── index.ts             # main(): decode → preflight → bypass → facts → eval → emit
-│       ├── bypass.ts            # build-reason auto-pass
-│       ├── facts.ts             # fact acquisition (env + REST)
-│       ├── predicates.ts        # 11 predicate evaluators + validatePredicateTree + glob ReDoS hardening
-│       └── selfcancel.ts        # best-effort build cancellation
+│   ├── gate/                    # gate.js entry point + per-concern modules
+│   │   ├── index.ts             # main(): decode → preflight → bypass → facts → eval → emit
+│   │   ├── bypass.ts            # build-reason auto-pass
+│   │   ├── facts.ts             # fact acquisition (env + REST)
+│   │   ├── predicates.ts        # 11 predicate evaluators + validatePredicateTree + glob ReDoS hardening
+│   │   └── selfcancel.ts        # best-effort build cancellation
+│   └── import/                  # import.js entry point + runtime prompt resolver
+│       ├── index.ts             # main(): expand runtime-import markers in place
+│       └── __tests__/           # marker, path-resolution, and single-pass coverage
 ├── test/                        # End-to-end smoke tests
-└── dist/gate/index.js           # ncc bundle output (gitignored)
+└── dist/                        # ncc bundle output (gitignored)
+    ├── gate/index.js
+    └── import/index.js
 ```
 
 The release workflow (`.github/workflows/release.yml`) runs
@@ -195,11 +228,18 @@ The Rust subcommand that emits the schema is intentionally hidden:
 cargo run -- export-gate-schema --output schema/gate-spec.schema.json
 ```
 
-## How the gate bundle is wired into emitted pipelines
+## How the bundles are wired into emitted pipelines
+
+`AdoScriptExtension`
+(`src/compile/extensions/ado_script.rs`) is the always-on single
+extension that owns all `ado-script` wiring. It has two independent
+features, each emitted **into the job that actually consumes the
+bundle**:
 
-`TriggerFiltersExtension`
-(`src/compile/extensions/trigger_filters.rs`) injects three Setup-job
-steps when any `filters:` block is active:
+### Setup job (gate evaluator)
+
+When `filters:` lowers to non-empty checks, `setup_steps()` returns
+three step strings into the Setup job:
 
 1. **`NodeTool@0`** — installs Node 20.x LTS, capped at
    `timeoutInMinutes: 5`.
@@ -209,9 +249,42 @@ steps when any `filters:` block is active:
    `unzip -o /tmp/ado-aw-scripts/ado-script.zip -d /tmp/ado-aw-scripts/`.
    Also capped at `timeoutInMinutes: 5`.
 3. **`bash: node '/tmp/ado-aw-scripts/ado-script/dist/gate/index.js'`** —
-   runs the gate with `GATE_SPEC` and the env-var contract above.
-
-The IR-to-bash codegen that produces these steps is
+   runs the gate with `GATE_SPEC` and the env-var contract documented
+   above.
+
+### Agent job (runtime-import resolver)
+
+When `inlined-imports: false` (the default), `prepare_steps()` returns
+the same install + download pair plus the resolver invocation, into
+the Agent job's existing `{{ prepare_steps }}` block:
+
+1. **`NodeTool@0`** — same shape as above.
+2. **`curl` download + verify + extract** — same artefact, same
+   verification.
+3. **`bash: node '/tmp/ado-aw-scripts/ado-script/dist/import/index.js'`** —
+   expands `{{#runtime-import …}}` markers in
+   `/tmp/awf-tools/agent-prompt.md` in place. See
+   [`runtime-imports.md`](runtime-imports.md) for marker syntax.
+
+### Per-job download (NOT a duplication bug)
+
+ADO jobs use **isolated VMs** — `/tmp` is not shared between jobs.
+The `ado-script.zip` bundle therefore has to be downloaded once per
+job that consumes it. When both features are active (a pipeline with
+both `filters:` and `inlined-imports: false`), install + download
+steps appear in **both** Setup and Agent. That's correct architecture
+given ADO's topology, not waste.
+
+### What gets emitted, by case
+
+| `filters:` | `inlined-imports` | Setup-job steps | Agent-job extra steps |
+|---|---|---|---|
+| inactive   | `true`  | (none)                              | (none)                              |
+| inactive   | `false` | (no Setup job)                      | install + download + resolver       |
+| active     | `true`  | install + download + gate           | (none)                              |
+| active     | `false` | install + download + gate           | install + download + resolver       |
+
+The IR-to-bash codegen that produces the gate step is
 `compile_gate_step_external` in `src/compile/filter_ir.rs`.
 
 ## Modifying `ado-script`

docs/filter-ir.md

@@ -350,24 +350,30 @@ The bash shim exports only the ADO macros needed by the spec's facts:
 
 ## Integration Points
 
-### TriggerFiltersExtension
+### AdoScriptExtension
 
-When Tier 2/3 filters are configured, the `TriggerFiltersExtension`
-(`src/compile/extensions/trigger_filters.rs`) activates via
-`collect_extensions()`. It implements `CompilerExtension` and controls:
+When `filters:` is configured (and lowers to non-empty checks), the
+always-on `AdoScriptExtension`
+(`src/compile/extensions/ado_script.rs`) emits the gate-side steps via
+the `setup_steps()` trait hook. The extension also owns the unrelated
+runtime-import resolver — see [`runtime-imports.md`](runtime-imports.md).
+
+For the gate path it controls:
 
 1. **Node install step** — emits a `NodeTool@0` step pinned to Node 20.x
-   LTS so `gate.js` has a runtime
+   LTS so `gate.js` has a runtime.
 2. **Download step** — fetches `ado-script.zip` from the ado-aw release
    artifacts, verifies its SHA256 checksum via `checksums.txt`, then
-   extracts `gate.js` to `/tmp/ado-aw-scripts/ado-script/dist/gate/index.js`
+   extracts `gate.js` to `/tmp/ado-aw-scripts/ado-script/dist/gate/index.js`.
 3. **Gate step** — calls `compile_gate_step_external()` to generate a step
-   that runs `node /tmp/ado-aw-scripts/ado-script/dist/gate/index.js` (no inline heredoc)
+   that runs `node /tmp/ado-aw-scripts/ado-script/dist/gate/index.js` (no inline heredoc).
 4. **Validation** — runs `validate_pr_filters()` / `validate_pipeline_filters()`
-   during compilation via the `validate()` trait method
+   during compilation via the `validate()` trait method.
 
-The extension uses the `setup_steps()` trait method (not `prepare_steps()`)
-because the gate must run in the **Setup job** (before the Agent job).
+The gate-side steps use `setup_steps()` (not `prepare_steps()`)
+because the gate must run in the **Setup job**, before the Agent job.
+Runtime-import resolver steps for the agent body use `prepare_steps()` and
+land in the Agent job — see [`runtime-imports.md`](runtime-imports.md).
 
 ### Tier 1 Inline Path
 
@@ -379,7 +385,7 @@ no Node evaluator and no download step.
 ### Gate Step Injection
 
 Gate steps are injected into the Setup job by `generate_setup_job()` in
-`common.rs`. When the `TriggerFiltersExtension` is active, its
+`common.rs`. When the `AdoScriptExtension` is active, its
 `setup_steps()` are collected and injected first (download + gate). When
 only Tier 1 filters are present, the inline gate step is injected directly.
 

docs/front-matter.md

@@ -234,6 +234,27 @@ becomes a `repos:` entry, with `checkout: false` added for entries
 that weren't listed under `checkout:`. Mixing the legacy fields with
 an existing `repos:` block is rejected; pick one shape.
 
+## Inlined Imports
+
+The `inlined-imports:` field controls when `{{#runtime-import ...}}`
+markers in the markdown body are resolved. It defaults to `false`.
+See [`runtime-imports.md`](runtime-imports.md) for the full marker
+syntax, path resolution rules, and runtime behavior.
+
+When `inlined-imports: false`, the compiler leaves runtime-import
+markers to be resolved on the pipeline runner. This is the default
+behavior, and it means prompt-body edits do not require recompiling the
+generated YAML.
+
+When `inlined-imports: true`, the compiler resolves all runtime-import
+markers at compile time, including the implicit top-level marker that
+normally reloads the body itself. The emitted YAML contains the fully
+expanded prompt body, so the pipeline file is self-contained.
+
+The trade-off is that the generated YAML is larger, and prompt-body
+edits require `ado-aw compile` plus committing the updated pipeline
+file.
+
 ## Filter Validation
 
 The compiler validates filter configurations at compile time and will emit

docs/runtime-imports.md

@@ -0,0 +1,82 @@
+# Runtime Imports
+
+_Part of the [ado-aw documentation](../AGENTS.md)._
+
+Runtime imports let agent prompts pull in snippet files with gh-aw-compatible
+`{{#runtime-import ...}}` markers. They are available in the markdown body and
+are controlled by the [`inlined-imports:` field](front-matter.md#inlined-imports).
+The runtime bundle that expands them is documented in [`ado-script.md`](ado-script.md).
+
+## Marker syntax
+
+Use `{{#runtime-import path}}` for a required import. If the target file is
+missing, resolution fails.
+
+```markdown
+## Repository policy
+{{#runtime-import docs/policy.md}}
+```
+
+Use `{{#runtime-import? path}}` for an optional import. If the target file is
+missing, the marker is replaced with an empty string.
+
+```markdown
+## Local notes
+{{#runtime-import? docs/local-notes.md}}
+```
+
+## Where markers can appear
+
+Authors can place runtime-import markers anywhere in the agent markdown body.
+When `inlined-imports: false` (the default), the compiler also injects an
+implicit top-level runtime-import marker that reloads the body itself at
+pipeline runtime instead of embedding it into the generated YAML. When
+`inlined-imports: true`, that implicit body marker is resolved at compile time
+along with any author-written markers.
+
+## Path resolution
+
+- **Absolute paths** are used as-is.
+- **Relative paths at runtime** are resolved against the `ADO_AW_IMPORT_BASE`
+  environment variable. For user-facing imports, the compiler sets this to
+  `{{ trigger_repo_directory }}`.
+- **Relative paths at compile time** (`inlined-imports: true`) are resolved
+  against the source `.md` file's directory.
+
+## Single-pass behavior
+
+Runtime imports are expanded in a single pass. Imported snippets are inserted
+verbatim, and any nested `{{#runtime-import ...}}` or
+`{{#runtime-import? ...}}` markers inside those snippets are **not** expanded.
+This matches gh-aw's runtime-import behavior.
+
+## Resolver ordering
+
+The runtime-import resolver runs first. Any extension supplements that are
+appended later with `cat >>` — including SafeOutputs guidance, GitHub MCP
+guidance, runtime guidance, and cache-memory guidance — are added after import
+resolution and are left untouched.
+
+## Failure modes
+
+| Marker kind | Missing file behavior |
+|---|---|
+| `{{#runtime-import path}}` | Resolver exits with status 1 and the pipeline fails. |
+| `{{#runtime-import? path}}` | Marker is silently replaced with an empty string. |
+
+When `inlined-imports: true`, the same required/optional rules are applied at
+compile time instead of on the pipeline runner.
+
+## Implementation notes
+
+- **Runtime**: `dist/import/index.js` is ncc-bundled into `ado-script.zip`.
+  The always-on `AdoScriptExtension`'s `prepare_steps()` injects three
+  steps into the Agent job's existing `{{ prepare_steps }}` block:
+  `NodeTool@0` install, the `ado-script.zip` download/verify/extract,
+  and the `node import.js` resolver invocation. All three run on the
+  same VM as the agent — ADO jobs are VM-isolated, so the bundle must
+  be downloaded inside whichever job consumes it.
+- **Compile time**: `resolve_imports_inline()` in
+  `src/compile/extensions/ado_script.rs` performs the inline expansion
+  when `inlined-imports: true`.
+

docs/template-markers.md

@@ -385,6 +385,8 @@ resources:
 
 Should be replaced with the markdown body (agent instructions) extracted from the source markdown file, excluding the YAML front matter. This content provides the agent with its task description and guidelines.
 
+When `inlined-imports: false` (the default), the compiler emits a top-level `{{#runtime-import ...}}` marker here so the prompt body is reloaded from the source markdown at pipeline runtime. When `inlined-imports: true`, any `{{#runtime-import ...}}` markers in the markdown body are resolved at compile time and the emitted YAML contains the expanded content directly.
+
 ## {{ mcpg_config }}
 
 Should be replaced with the MCP Gateway (MCPG) configuration JSON generated from the `mcp-servers:` front matter. This configuration defines the MCPG server entries and gateway settings.
@@ -487,6 +489,8 @@ Tool names are validated at compile time:
 
 Should be replaced with the embedded threat detection analysis prompt from `src/data/threat-analysis.md`. This prompt template includes markers for `{{ source_path }}`, `{{ agent_name }}`, `{{ agent_description }}`, and `{{ working_directory }}` which are replaced during compilation.
 
+When `inlined-imports: false`, the compiler emits a top-level `{{#runtime-import ...}}` marker pointing at the bundled threat-analysis prompt that ships in `ado-script.zip`; when `inlined-imports: true`, the expanded prompt body is embedded directly into the YAML.
+
 The threat analysis prompt instructs the security analysis agent to check for:
 - Prompt injection attempts
 - Secret leaks