Skip to content

Commit d178081

Browse files
jamesadevineCopilot
andcommitted
feat(compile): add target: job and target: stage for ADO template output
Add two new compile targets that produce reusable ADO YAML templates for embedding agentic stages into existing pipelines: - target: job — generates a job-level template (jobs: at root) that can be included in a flat pipeline or inside a user-defined stage - target: stage — generates a stage-level template (stages: wrapping jobs) for direct inclusion in multi-stage pipelines Key design decisions: - Pool is baked in from front matter (not a template parameter) - dependsOn and condition are set natively at the ADO call site - Job names are prefixed with PascalCase agent name for uniqueness (e.g., DailyReview_Agent, DailyReview_Detection, DailyReview_Execution) - Triggers (on:) are ignored with a warning in template targets - Template parameters only include clearMemory and user-defined params New files: - src/compile/job.rs — JobCompiler implementing the Compiler trait - src/compile/stage.rs — StageCompiler implementing the Compiler trait - src/data/job-base.yml — job-level template derived from base.yml - src/data/stage-base.yml — stage-level template wrapping jobs in stage Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent ee7080c commit d178081

16 files changed

Lines changed: 1857 additions & 7 deletions

AGENTS.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,8 @@ Every compiled pipeline runs as three sequential jobs:
5252
│ │ ├── common.rs # Shared helpers across targets
5353
│ │ ├── standalone.rs # Standalone pipeline compiler
5454
│ │ ├── onees.rs # 1ES Pipeline Template compiler
55+
│ │ ├── job.rs # Job-level ADO template compiler (target: job)
56+
│ │ ├── stage.rs # Stage-level ADO template compiler (target: stage)
5557
│ │ ├── gitattributes.rs # .gitattributes management for compiled pipelines
5658
│ │ ├── filter_ir.rs # Filter expression IR: Fact/Predicate types, lowering, validation, codegen
5759
│ │ ├── pr_filters.rs # PR trigger filter generation (native ADO + gate steps)
@@ -122,6 +124,8 @@ Every compiled pipeline runs as three sequential jobs:
122124
│ ├── data/
123125
│ │ ├── base.yml # Base pipeline template for standalone
124126
│ │ ├── 1es-base.yml # Base pipeline template for 1ES target
127+
│ │ ├── job-base.yml # Job-level ADO template for target: job
128+
│ │ ├── stage-base.yml # Stage-level ADO template for target: stage
125129
│ │ ├── ecosystem_domains.json # Network allowlists per ecosystem
126130
│ │ ├── init-agent.md # Dispatcher agent template for `init` command
127131
│ │ └── threat-analysis.md # Threat detection analysis prompt template

docs/cli.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ Global flags (apply to all subcommands): `--verbose, -v` (enable info-level logg
1515
- `--output, -o <path>` - Optional output path for the generated YAML (only valid when a path is provided). If the path is an existing directory, the compiled YAML is written inside that directory using the default filename derived from the markdown source (e.g. `foo.md``<dir>/foo.lock.yml`).
1616
- `--skip-integrity` - *(debug builds only)* Omit the "Verify pipeline integrity" step from the generated pipeline. Useful during local development when the compiled output won't match a released compiler version. This flag is not available in release builds.
1717
- `--debug-pipeline` - *(debug builds only)* Include MCPG debug diagnostics in the generated pipeline: `DEBUG=*` environment variable for verbose MCPG logging, stderr streaming to log files, and a "Verify MCP backends" step that probes each backend with MCP initialize + tools/list before the agent runs. This flag is not available in release builds.
18+
- For `target: job` and `target: stage`, the output is an ADO YAML template (not a complete pipeline). Job names are prefixed with the agent name for uniqueness. Triggers configured via `on:` are ignored with a warning.
1819
- `check <pipeline>` - Verify that a compiled pipeline matches its source markdown
1920
- `<pipeline>` - Path to the pipeline YAML file to verify
2021
- The source markdown path is auto-detected from the `@ado-aw` header in the pipeline file

docs/front-matter.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ The compiler expects markdown files with YAML front matter similar to gh-aw:
1010
---
1111
name: "name for this agent"
1212
description: "One line description for this agent"
13-
target: standalone # Optional: "standalone" (default) or "1es". See docs/targets.md.
13+
target: standalone # Optional: "standalone" (default), "1es", "job", or "stage". See docs/targets.md.
1414
engine: copilot # Engine identifier. Defaults to copilot. Currently only 'copilot' (GitHub Copilot CLI) is supported.
1515
# engine: # Alternative object format (with additional options)
1616
# id: copilot

docs/targets.md

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,3 +31,79 @@ target: 1es
3131
```
3232
3333
When using `target: 1es`, the pipeline will extend `1es/1ES.Unofficial.PipelineTemplate.yml@1ESPipelinesTemplates`.
34+
35+
### `job`
36+
37+
Generates a **job-level ADO YAML template** with `jobs:` at root. This is a
38+
reusable template that can be included in an existing pipeline — it does not
39+
generate a complete pipeline.
40+
41+
The output contains the same 3-job chain (Agent → Detection → Execution) as
42+
`standalone`, with:
43+
- Job names prefixed with the agent name for uniqueness (e.g., `DailyReview_Agent`)
44+
- No triggers, pipeline name, or resource declarations (the parent pipeline owns those)
45+
- Pool baked in from the front matter `pool:` field
46+
47+
Example front matter:
48+
```yaml
49+
target: job
50+
```
51+
52+
#### Usage in a flat pipeline
53+
54+
```yaml
55+
jobs:
56+
- job: Build
57+
steps: ...
58+
- template: agents/review.lock.yml
59+
```
60+
61+
#### Usage inside a user-defined stage
62+
63+
```yaml
64+
stages:
65+
- stage: Build
66+
jobs: ...
67+
- stage: AgenticReview
68+
dependsOn: Build
69+
jobs:
70+
- template: agents/review.lock.yml
71+
```
72+
73+
#### Notes
74+
75+
- Triggers (`on:`) are ignored with a warning (the parent pipeline controls triggers)
76+
- If the agent declares additional repositories via `repos:`, add them to the
77+
parent pipeline's `resources:` block (documented in the generated file header)
78+
79+
### `stage`
80+
81+
Generates a **stage-level ADO YAML template** with `stages:` at root. This
82+
wraps the 3-job chain inside a stage block for direct inclusion in multi-stage
83+
pipelines.
84+
85+
Example front matter:
86+
```yaml
87+
target: stage
88+
```
89+
90+
#### Usage
91+
92+
```yaml
93+
stages:
94+
- stage: Build
95+
jobs: ...
96+
- template: agents/review.lock.yml
97+
dependsOn: Build
98+
condition: succeeded()
99+
```
100+
101+
ADO natively supports `dependsOn` and `condition` at the template call site —
102+
no template parameters are needed for stage ordering.
103+
104+
#### Notes
105+
106+
- Same 3-job chain, job-name prefixing, and pool handling as `target: job`
107+
- Triggers (`on:`) are ignored with a warning
108+
- If the agent declares additional repositories via `repos:`, add them to the
109+
parent pipeline's `resources:` block

src/compile/common.rs

Lines changed: 64 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -999,6 +999,59 @@ pub fn sanitize_filename(name: &str) -> String {
999999
/// Default pool name
10001000
pub const DEFAULT_POOL: &str = "AZS-1ES-L-MMS-ubuntu-22.04";
10011001

1002+
/// Derive a valid ADO identifier from the agent name for use as a job-name
1003+
/// prefix and stage name. Converts to PascalCase, stripping non-alphanumeric
1004+
/// characters.
1005+
///
1006+
/// Examples:
1007+
/// - `"Daily Code Review"` → `"DailyCodeReview"`
1008+
/// - `"my-agent-123"` → `"MyAgent123"`
1009+
/// - `""` → `"Agent"` (fallback)
1010+
/// - `"123start"` → `"_123start"` (prefix underscore for leading digit)
1011+
pub fn generate_stage_prefix(name: &str) -> String {
1012+
let pascal: String = name
1013+
.split(|c: char| !c.is_alphanumeric())
1014+
.filter(|s| !s.is_empty())
1015+
.map(|word| {
1016+
let mut chars = word.chars();
1017+
match chars.next() {
1018+
None => String::new(),
1019+
Some(first) => {
1020+
let upper = first.to_uppercase().to_string();
1021+
upper + chars.as_str()
1022+
}
1023+
}
1024+
})
1025+
.collect();
1026+
1027+
if pascal.is_empty() {
1028+
"Agent".to_string()
1029+
} else if pascal.starts_with(|c: char| c.is_ascii_digit()) {
1030+
format!("_{}", pascal)
1031+
} else {
1032+
pascal
1033+
}
1034+
}
1035+
1036+
/// Generate the template-level `parameters:` YAML block for job/stage
1037+
/// template targets.
1038+
///
1039+
/// Includes clearMemory (if cache-memory enabled) and user-defined
1040+
/// parameters from front matter. Returns empty string if no parameters
1041+
/// are needed.
1042+
pub fn generate_template_parameters(front_matter: &FrontMatter) -> Result<String> {
1043+
let has_memory = front_matter
1044+
.tools
1045+
.as_ref()
1046+
.and_then(|t| t.cache_memory.as_ref())
1047+
.is_some_and(|cm| cm.is_enabled());
1048+
let params = build_parameters(&front_matter.parameters, has_memory);
1049+
if params.is_empty() {
1050+
return Ok(String::new());
1051+
}
1052+
generate_parameters(&params)
1053+
}
1054+
10021055
/// Version of the AWF (Agentic Workflow Firewall) binary to download from GitHub Releases.
10031056
/// Update this when upgrading to a new AWF release.
10041057
/// See: https://github.com/github/gh-aw-firewall/releases
@@ -2423,6 +2476,10 @@ pub struct CompileConfig {
24232476
/// to append `GITHUB_PATH: $(GITHUB_PATH)` to the engine env block without
24242477
/// re-collecting path prepends from extensions.
24252478
pub has_awf_paths: bool,
2479+
/// When true, `compile_shared` omits the standard `# @ado-aw` header.
2480+
/// Template-producing compilers (Job, Stage) set this to prepend their
2481+
/// own custom header with usage instructions.
2482+
pub skip_header: bool,
24262483
}
24272484

24282485
/// Shared compilation flow used by both standalone and 1ES compilers.
@@ -2704,9 +2761,13 @@ pub async fn compile_shared(
27042761
replace_with_indent(&yaml, placeholder, replacement)
27052762
});
27062763

2707-
// 15. Prepend header
2708-
let header = generate_header_comment(input_path);
2709-
Ok(format!("{}{}", header, pipeline_yaml))
2764+
// 15. Prepend header (unless the caller will prepend its own)
2765+
if config.skip_header {
2766+
Ok(pipeline_yaml)
2767+
} else {
2768+
let header = generate_header_comment(input_path);
2769+
Ok(format!("{}{}", header, pipeline_yaml))
2770+
}
27102771
}
27112772

27122773
#[cfg(test)]

src/compile/job.rs

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
//! Job-level ADO template compiler.
2+
//!
3+
//! This compiler generates a reusable ADO YAML template with `jobs:` at root.
4+
//! Users include it in their existing pipelines via `- template: <path>`.
5+
//!
6+
//! Two inclusion patterns:
7+
//! - Directly in a flat pipeline's `jobs:` list
8+
//! - Inside a user-defined stage's `jobs:` list
9+
10+
use anyhow::{Context, Result};
11+
use async_trait::async_trait;
12+
use log::{info, warn};
13+
use std::path::Path;
14+
15+
use super::Compiler;
16+
use super::common::{
17+
AWF_VERSION, MCPG_VERSION, MCPG_IMAGE, MCPG_PORT, MCPG_DOMAIN,
18+
CompileConfig, compile_shared,
19+
generate_allowed_domains,
20+
generate_awf_mounts,
21+
generate_awf_path_step,
22+
collect_awf_path_prepends,
23+
generate_enabled_tools_args,
24+
generate_mcpg_config, generate_mcpg_docker_env, generate_mcpg_step_env,
25+
generate_stage_prefix, generate_template_parameters,
26+
generate_header_comment,
27+
};
28+
use super::types::FrontMatter;
29+
30+
/// Job-level template compiler.
31+
pub struct JobCompiler;
32+
33+
#[async_trait]
34+
impl Compiler for JobCompiler {
35+
fn target_name(&self) -> &'static str {
36+
"job"
37+
}
38+
39+
async fn compile(
40+
&self,
41+
input_path: &Path,
42+
output_path: &Path,
43+
front_matter: &FrontMatter,
44+
markdown_body: &str,
45+
skip_integrity: bool,
46+
debug_pipeline: bool,
47+
) -> Result<String> {
48+
info!("Compiling for job template target");
49+
50+
if front_matter.on_config.is_some() {
51+
warn!("on: trigger configuration is ignored for target: job (triggers are the parent pipeline's concern)");
52+
}
53+
54+
// Collect extensions (needed before compile_shared for MCPG config)
55+
let extensions = super::extensions::collect_extensions(front_matter);
56+
57+
// Build compile context for MCPG config generation
58+
let input_dir = input_path.parent().unwrap_or(std::path::Path::new("."));
59+
let ctx = super::extensions::CompileContext::new(front_matter, input_dir).await?;
60+
61+
// Generate stage prefix for job-name uniqueness
62+
let stage_prefix = generate_stage_prefix(&front_matter.name);
63+
64+
// Generate template-level parameters
65+
let template_params = generate_template_parameters(front_matter)?;
66+
67+
// Same AWF/MCPG values as standalone
68+
let allowed_domains = generate_allowed_domains(front_matter, &extensions)?;
69+
let awf_mounts = generate_awf_mounts(&extensions);
70+
let awf_paths = collect_awf_path_prepends(&extensions);
71+
let awf_path_step = generate_awf_path_step(&awf_paths);
72+
let enabled_tools_args = generate_enabled_tools_args(front_matter);
73+
74+
let config_obj = generate_mcpg_config(front_matter, &ctx, &extensions)?;
75+
let mcpg_config_json =
76+
serde_json::to_string_pretty(&config_obj).context("Failed to serialize MCPG config")?;
77+
let mcpg_docker_env = generate_mcpg_docker_env(front_matter, &extensions);
78+
let mcpg_step_env = generate_mcpg_step_env(&extensions);
79+
80+
let config = CompileConfig {
81+
template: include_str!("../data/job-base.yml").to_string(),
82+
extra_replacements: vec![
83+
("{{ stage_prefix }}".into(), stage_prefix),
84+
("{{ template_parameters }}".into(), template_params),
85+
("{{ firewall_version }}".into(), AWF_VERSION.into()),
86+
("{{ mcpg_version }}".into(), MCPG_VERSION.into()),
87+
("{{ mcpg_image }}".into(), MCPG_IMAGE.into()),
88+
("{{ mcpg_port }}".into(), MCPG_PORT.to_string()),
89+
("{{ mcpg_domain }}".into(), MCPG_DOMAIN.into()),
90+
("{{ allowed_domains }}".into(), allowed_domains),
91+
("{{ awf_mounts }}".into(), awf_mounts),
92+
("{{ awf_path_step }}".into(), awf_path_step),
93+
("{{ enabled_tools_args }}".into(), enabled_tools_args),
94+
("{{ mcpg_config }}".into(), mcpg_config_json),
95+
("{{ mcpg_docker_env }}".into(), mcpg_docker_env),
96+
("{{ mcpg_step_env }}".into(), mcpg_step_env),
97+
],
98+
skip_integrity,
99+
debug_pipeline,
100+
has_awf_paths: !awf_paths.is_empty(),
101+
skip_header: true,
102+
};
103+
104+
let yaml = compile_shared(
105+
input_path, output_path, front_matter, markdown_body,
106+
&extensions, &ctx, config,
107+
).await?;
108+
109+
// Prepend custom header with job-template usage instructions
110+
let header = generate_job_header(input_path, front_matter);
111+
Ok(format!("{}{}", header, yaml))
112+
}
113+
}
114+
115+
/// Generate the header comment block for job-level templates.
116+
fn generate_job_header(input_path: &Path, front_matter: &FrontMatter) -> String {
117+
let base_header = generate_header_comment(input_path);
118+
let source_path = input_path
119+
.to_string_lossy()
120+
.replace('\\', "/");
121+
122+
let mut header = base_header;
123+
header.push_str("#\n");
124+
header.push_str("# Job-level ADO template. Include in your pipeline:\n");
125+
header.push_str("#\n");
126+
header.push_str(&format!("# jobs:\n"));
127+
header.push_str(&format!("# - template: {}\n", source_path.replace(".md", ".lock.yml")));
128+
header.push_str("#\n");
129+
header.push_str("# Or inside a stage in a multi-stage pipeline:\n");
130+
header.push_str("#\n");
131+
header.push_str("# stages:\n");
132+
header.push_str("# - stage: AgenticReview\n");
133+
header.push_str("# dependsOn: Build\n");
134+
header.push_str("# jobs:\n");
135+
header.push_str(&format!("# - template: {}\n", source_path.replace(".md", ".lock.yml")));
136+
137+
// Document required resources if agent uses repos
138+
if !front_matter.repos.is_empty() || !front_matter.repositories.is_empty() {
139+
header.push_str("#\n");
140+
header.push_str("# Add these repositories to your pipeline's resources: block:\n");
141+
header.push_str("#\n");
142+
header.push_str("# resources:\n");
143+
header.push_str("# repositories:\n");
144+
for repo in &front_matter.repositories {
145+
header.push_str(&format!("# - repository: {}\n", repo.repository));
146+
header.push_str(&format!("# type: {}\n", repo.repo_type));
147+
header.push_str(&format!("# name: {}\n", repo.name));
148+
}
149+
}
150+
151+
header.push('\n');
152+
header
153+
}
154+
155+
#[cfg(test)]
156+
mod tests {
157+
use super::*;
158+
use crate::compile::common::generate_stage_prefix;
159+
160+
#[test]
161+
fn test_generate_stage_prefix_basic() {
162+
assert_eq!(generate_stage_prefix("Daily Code Review"), "DailyCodeReview");
163+
}
164+
165+
#[test]
166+
fn test_generate_stage_prefix_hyphens() {
167+
assert_eq!(generate_stage_prefix("my-agent-123"), "MyAgent123");
168+
}
169+
170+
#[test]
171+
fn test_generate_stage_prefix_empty() {
172+
assert_eq!(generate_stage_prefix(""), "Agent");
173+
}
174+
175+
#[test]
176+
fn test_generate_stage_prefix_leading_digit() {
177+
assert_eq!(generate_stage_prefix("123start"), "_123start");
178+
}
179+
180+
#[test]
181+
fn test_generate_stage_prefix_single_word() {
182+
assert_eq!(generate_stage_prefix("review"), "Review");
183+
}
184+
185+
#[test]
186+
fn test_generate_stage_prefix_underscores() {
187+
assert_eq!(generate_stage_prefix("code_review_agent"), "CodeReviewAgent");
188+
}
189+
}

0 commit comments

Comments
 (0)