Skip to content

Commit 39103e1

Browse files
feat: promote memory to cache-memory tool and add first-class azure-devops tool (#167)
* refactor: rename src/tools to src/safeoutputs, create src/tools for first-class tools Rename the existing tools directory to safeoutputs to better reflect its purpose (safe-output MCP tool implementations that serialize to NDJSON in Stage 1 and execute in Stage 2). Create a new src/tools directory for first-class tool implementations that the compiler auto-configures (cache-memory, azure-devops). Move memory.rs from safeoutputs to tools/cache_memory.rs since memory is a first-class tool, not a safe-output. Add CacheMemoryToolConfig and AzureDevOpsToolConfig types to compile/types.rs with support for both boolean and object front-matter formats. Extend ToolsConfig to include cache-memory and azure-devops fields. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: migrate memory from safe-outputs to tools.cache-memory Move memory configuration from safe-outputs: memory: to tools: cache-memory: in the front matter. This aligns with gh-aw's cache-memory tool pattern where memory is a first-class tool, not a safe-output. Key changes: - Update has_memory detection in standalone.rs and onees.rs to read from tools.cache-memory instead of safe-outputs.memory - Update main.rs Stage 2 executor to resolve MemoryConfig from tools.cache-memory - Remove 'memory' from NON_MCP_SAFE_OUTPUT_KEYS and ALL_KNOWN_SAFE_OUTPUTS - Update integration tests to use tools: cache-memory: format - Update enabled-tools-args tests (memory no longer affects filtering) - No backward compatibility for safe-outputs.memory Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add first-class azure-devops tool support Add tools.azure-devops as a first-class compiler tool that auto-configures the Azure DevOps MCP container in the MCPG config. This replaces the need for manual mcp-servers configuration with boilerplate container/entrypoint settings. When tools.azure-devops is enabled, the compiler: - Auto-generates a containerized stdio MCP entry (node:20-slim + npx @azure-devops/mcp) in the MCPG configuration - Auto-maps ADO token (AZURE_DEVOPS_EXT_PAT) passthrough when permissions.read is configured - Adds ADO-specific hosts to the network allowlist - Supports toolsets (repos, wit, core etc.) as -d flags - Supports explicit tool allow-list for MCPG filtering - Auto-infers org from pipeline runtime variables with optional override - Warns on conflict with manual mcp-servers.azure-devops entry Front-matter example: tools: azure-devops: toolsets: [repos, wit] allowed: [wit_get_work_item] org: myorg # optional, auto-inferred Also adds ADO_ORG_NAME runtime extraction to the base template for org auto-inference from $(System.TeamFoundationCollectionUri). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: update AGENTS.md and examples for tools refactor - Update architecture tree to show src/safeoutputs/ and src/tools/ - Add cache-memory and azure-devops tool documentation under Tools Configuration - Update memory safe-output section to point to new tools.cache-memory location - Update front-matter example to show new tool entries - Update 'Adding New Features' section with safeoutputs vs tools distinction - Update azure-devops-mcp.md example to use tools.azure-devops instead of manual mcp-servers configuration Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: infer ADO org from git remote at compile time, fail on missing org Replace the runtime $(ADO_ORG_NAME) pipeline variable approach with compile-time inference using the existing parse_ado_remote() function. The compiler now extracts the org from the git remote URL when compiling. Key changes: - generate_mcpg_config() now returns Result and accepts inferred_org - Compilation fails with a clear error if tools.azure-devops is enabled but no org can be determined (no explicit override + no ADO git remote) - Remove $(ADO_ORG_NAME) runtime substitution from base.yml template - Remove unused project field from AzureDevOpsOptions (not supported by @azure-devops/mcp and was silently discarded) - Make get_git_remote_url public for use by the compiler - Add test_ado_tool_no_org_fails and test_ado_tool_explicit_org_overrides_inferred Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: remove project field — not supported by @azure-devops/mcp The ADO MCP only accepts org as a positional arg. The project field was added speculatively but is not a supported option. Keep only org (with compile-time git remote inference and explicit override). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: wire MCPG_IMAGE constant into template, fix stale memory comment - Replace hardcoded ghcr.io/github/gh-aw-mcpg in base.yml with {{ mcpg_image }} marker, replaced by MCPG_IMAGE constant in standalone.rs — single source of truth for the image name - Fix stale comment in mcp.rs that still referenced 'memory' in NON_MCP_SAFE_OUTPUT_KEYS (now empty) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: validate ADO org/toolset names, add memory deprecation hint - Validate org name (alphanumerics + hyphens only) at compile time to catch invalid values early instead of cryptic MCPG runtime failures - Validate toolset names with the same rule - Add specific deprecation hint when safe-outputs: memory: is detected, directing users to tools: cache-memory: - Add tests for invalid org and invalid toolset rejection Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 6700677 commit 39103e1

38 files changed

Lines changed: 1320 additions & 600 deletions

AGENTS.md

Lines changed: 86 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -37,19 +37,33 @@ Alongside the correctly generated pipeline yaml, an agent file is generated from
3737
│ ├── ndjson.rs # NDJSON parsing utilities
3838
│ ├── proxy.rs # Network proxy implementation
3939
│ ├── sanitize.rs # Input sanitization for safe outputs
40-
│ └── tools/ # MCP tool implementations
40+
│ ├── safeoutputs/ # Safe-output MCP tool implementations (Stage 1 → NDJSON → Stage 2)
41+
│ │ ├── mod.rs
42+
│ │ ├── add_build_tag.rs
43+
│ │ ├── add_pr_comment.rs
44+
│ │ ├── comment_on_work_item.rs
45+
│ │ ├── create_branch.rs
46+
│ │ ├── create_git_tag.rs
47+
│ │ ├── create_pr.rs
48+
│ │ ├── create_wiki_page.rs
49+
│ │ ├── create_work_item.rs
50+
│ │ ├── link_work_items.rs
51+
│ │ ├── missing_data.rs
52+
│ │ ├── missing_tool.rs
53+
│ │ ├── noop.rs
54+
│ │ ├── queue_build.rs
55+
│ │ ├── reply_to_pr_comment.rs
56+
│ │ ├── report_incomplete.rs
57+
│ │ ├── resolve_pr_thread.rs
58+
│ │ ├── result.rs
59+
│ │ ├── submit_pr_review.rs
60+
│ │ ├── update_pr.rs
61+
│ │ ├── update_wiki_page.rs
62+
│ │ ├── update_work_item.rs
63+
│ │ └── upload_attachment.rs
64+
│ └── tools/ # First-class tool implementations (compiler auto-configures)
4165
│ ├── mod.rs
42-
│ ├── comment_on_work_item.rs
43-
│ ├── create_pr.rs
44-
│ ├── create_wiki_page.rs
45-
│ ├── create_work_item.rs
46-
│ ├── update_work_item.rs
47-
│ ├── update_wiki_page.rs
48-
│ ├── memory.rs
49-
│ ├── missing_data.rs
50-
│ ├── missing_tool.rs
51-
│ ├── noop.rs
52-
│ └── result.rs
66+
│ └── cache_memory.rs
5367
├── templates/
5468
│ ├── base.yml # Base pipeline template for standalone
5569
│ ├── 1es-base.yml # Base pipeline template for 1ES target
@@ -125,6 +139,14 @@ checkout: # optional list of repository aliases for the agent to checkout and wo
125139
tools: # optional tool configuration
126140
bash: ["cat", "ls", "grep"] # bash command allow-list (defaults to safe built-in list)
127141
edit: true # enable file editing tool (default: true)
142+
cache-memory: true # persistent memory across runs (see Cache Memory section)
143+
# cache-memory: # Alternative object format (with options)
144+
# allowed-extensions: [.md, .json]
145+
azure-devops: true # first-class ADO MCP integration (see Azure DevOps MCP section)
146+
# azure-devops: # Alternative object format (with scoping)
147+
# toolsets: [repos, wit]
148+
# allowed: [wit_get_work_item]
149+
# org: myorg
128150
# env: # RESERVED: workflow-level environment variables (not yet implemented)
129151
# CUSTOM_VAR: "value"
130152
mcp-servers:
@@ -392,6 +414,53 @@ tools:
392414
edit: false
393415
```
394416

417+
#### Cache Memory (`cache-memory:`)
418+
419+
Persistent memory storage across agent runs. The agent reads/writes files to a memory directory that persists between pipeline executions via pipeline artifacts.
420+
421+
```yaml
422+
# Simple enablement
423+
tools:
424+
cache-memory: true
425+
426+
# With options
427+
tools:
428+
cache-memory:
429+
allowed-extensions: [.md, .json, .txt]
430+
```
431+
432+
When enabled, the compiler auto-generates pipeline steps to:
433+
- Download previous memory from the last successful run's artifact
434+
- Restore files to `/tmp/awf-tools/staging/agent_memory/`
435+
- Append a memory prompt to the agent instructions
436+
- Auto-inject a `clearMemory` pipeline parameter (allows clearing memory from the ADO UI)
437+
438+
During Stage 2 execution, memory files are validated (path safety, extension filtering, `##vso[` injection detection, 5 MB size limit) and published as a pipeline artifact.
439+
440+
#### Azure DevOps MCP (`azure-devops:`)
441+
442+
First-class Azure DevOps MCP integration. Auto-configures the ADO MCP container, token mapping, MCPG entry, and network allowlist.
443+
444+
```yaml
445+
# Simple enablement (auto-infers org from git remote)
446+
tools:
447+
azure-devops: true
448+
449+
# With scoping options
450+
tools:
451+
azure-devops:
452+
toolsets: [repos, wit, core] # ADO API toolset groups
453+
allowed: [wit_get_work_item, core_list_projects] # Explicit tool allow-list
454+
org: myorg # Optional override (inferred from git remote)
455+
```
456+
457+
When enabled, the compiler:
458+
- Generates a containerized stdio MCP entry (`node:20-slim` + `npx @azure-devops/mcp`) in the MCPG config
459+
- Auto-maps `AZURE_DEVOPS_EXT_PAT` token passthrough when `permissions.read` is configured
460+
- Adds ADO-specific hosts to the network allowlist
461+
- Auto-infers org from the git remote URL at compile time (overridable via `org:` field)
462+
- Fails compilation if org cannot be determined (no explicit override and no ADO git remote)
463+
395464
### Target Platforms
396465

397466
The `target` field in the front matter determines the output format and execution environment for the compiled pipeline.
@@ -1066,35 +1135,8 @@ Reports that a tool or capability needed to complete the task is not available.
10661135
- `tool_name` - Name of the tool that was expected but not found
10671136
- `context` - Optional context about why the tool was needed
10681137

1069-
#### memory
1070-
Provides persistent memory across agent runs. When enabled, the agent can read and write files to a memory directory that persists between pipeline executions.
1071-
1072-
**Configuration options (front matter):**
1073-
```yaml
1074-
safe-outputs:
1075-
memory:
1076-
allowed-extensions: # Optional: restrict file types (defaults to all)
1077-
- .md
1078-
- .json
1079-
- .txt
1080-
```
1081-
1082-
**How it works:**
1083-
1. During Stage 1 (agent execution), the agent can write files to `/tmp/awf-tools/staging/agent_memory/`
1084-
2. A prompt is automatically appended to inform the agent about its memory location
1085-
3. During Stage 2 execution, memory files are validated and sanitized:
1086-
- Path traversal attempts are blocked
1087-
- Files are checked for `##vso[` command injection
1088-
- Total size is limited to 5 MB
1089-
- File extensions can be restricted via configuration
1090-
4. Sanitized memory files are published as a pipeline artifact
1091-
5. On the next run, the previous memory is downloaded and restored to the staging directory
1092-
1093-
**Security validations:**
1094-
- Maximum total memory size: 5 MB
1095-
- Path validation: no `..`, `.git`, absolute paths, or null bytes
1096-
- Content validation: text files are scanned for `##vso[` commands
1097-
- Extension filtering: can restrict to specific file types
1138+
#### cache-memory (moved to `tools:`)
1139+
Memory is now configured as a first-class tool under `tools: cache-memory:` instead of `safe-outputs: memory:`. See the [Cache Memory](#cache-memory-cache-memory) section under Tools Configuration for details.
10981140

10991141
#### create-wiki-page
11001142
Creates a new Azure DevOps wiki page. The page must **not** already exist; the tool enforces an atomic create-only operation (via `If-Match: ""`). Attempting to create a page that already exists results in an explicit failure.
@@ -1154,7 +1196,9 @@ When extending the compiler:
11541196
2. **New compile targets**: Implement the `Compiler` trait in a new file under `src/compile/`
11551197
3. **New front matter fields**: Add fields to `FrontMatter` in `src/compile/types.rs`
11561198
4. **New template markers**: Handle replacements in the target-specific compiler (e.g., `standalone.rs` or `onees.rs`)
1157-
5. **Validation**: Add compile-time validation for safe outputs and permissions
1199+
5. **New safe-output tools**: Add to `src/safeoutputs/` — implement `ToolResult`, `Executor`, register in `mod.rs`, `mcp.rs`, `execute.rs`
1200+
6. **New first-class tools**: Add to `src/tools/` — extend `ToolsConfig` in `types.rs`, wire in compilers
1201+
7. **Validation**: Add compile-time validation for safe outputs and permissions
11581202

11591203
### Security Considerations
11601204

examples/azure-devops-mcp.md

Lines changed: 2 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,9 @@ name: "ADO Work Item Triage"
33
description: "Triages work items using the Azure DevOps MCP"
44
schedule: daily around 9:00
55
engine: claude-sonnet-4.5
6-
mcp-servers:
6+
tools:
77
azure-devops:
8-
container: "node:20-slim"
9-
entrypoint: "npx"
10-
entrypoint-args: ["-y", "@azure-devops/mcp", "myorg", "-d", "core", "work", "work-items"]
11-
env:
12-
AZURE_DEVOPS_EXT_PAT: ""
8+
toolsets: [core, work, work-items]
139
allowed:
1410
- core_list_projects
1511
- wit_my_work_items
@@ -31,11 +27,6 @@ safe-outputs:
3127
comment-on-work-item:
3228
max: 10
3329
target: "*"
34-
network:
35-
allow:
36-
- "dev.azure.com"
37-
- "*.dev.azure.com"
38-
- "*.visualstudio.com"
3930
---
4031

4132
## ADO Work Item Triage Agent

src/compile/common.rs

Lines changed: 42 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -625,9 +625,25 @@ pub fn generate_header_comment(input_path: &std::path::Path) -> String {
625625
/// See: https://github.com/github/gh-aw-mcpg/releases
626626
pub const MCPG_VERSION: &str = "0.2.18";
627627

628+
/// Docker image for the MCPG container.
629+
pub const MCPG_IMAGE: &str = "ghcr.io/github/gh-aw-mcpg";
630+
628631
/// Default port MCPG listens on inside the container (host network mode).
629632
pub const MCPG_PORT: u16 = 80;
630633

634+
/// Docker image for the Azure DevOps MCP container.
635+
/// This is the container used when `tools: azure-devops:` is configured.
636+
pub const ADO_MCP_IMAGE: &str = "node:20-slim";
637+
638+
/// Default entrypoint for the Azure DevOps MCP container.
639+
pub const ADO_MCP_ENTRYPOINT: &str = "npx";
640+
641+
/// Default entrypoint args for the Azure DevOps MCP npm package.
642+
pub const ADO_MCP_PACKAGE: &str = "@azure-devops/mcp";
643+
644+
/// Reserved MCPG server name for the auto-configured ADO MCP.
645+
pub const ADO_MCP_SERVER_NAME: &str = "azure-devops";
646+
631647
/// Generate source path for the execute command.
632648
///
633649
/// Returns a path using `{{ workspace }}` as the base, which gets resolved
@@ -805,14 +821,11 @@ fn is_safe_tool_name(name: &str) -> bool {
805821
/// diagnostic tools. If `safe-outputs:` is empty, returns an empty string
806822
/// (all tools enabled for backward compatibility).
807823
///
808-
/// Non-MCP keys (like `memory`) are silently skipped — they are handled by the
809-
/// executor and have no corresponding MCP route.
810-
///
811824
/// Tool names are validated to contain only ASCII alphanumerics and hyphens
812825
/// to prevent shell injection when the args are embedded in bash commands.
813826
/// Unrecognized tool names emit a compile-time warning and are skipped.
814827
pub fn generate_enabled_tools_args(front_matter: &FrontMatter) -> String {
815-
use crate::tools::{ALL_KNOWN_SAFE_OUTPUTS, ALWAYS_ON_TOOLS, NON_MCP_SAFE_OUTPUT_KEYS};
828+
use crate::safeoutputs::{ALL_KNOWN_SAFE_OUTPUTS, ALWAYS_ON_TOOLS, NON_MCP_SAFE_OUTPUT_KEYS};
816829
use std::collections::HashSet;
817830

818831
if front_matter.safe_outputs.is_empty() {
@@ -835,6 +848,14 @@ pub fn generate_enabled_tools_args(front_matter: &FrontMatter) -> String {
835848
if NON_MCP_SAFE_OUTPUT_KEYS.contains(&key.as_str()) {
836849
continue;
837850
}
851+
if key == "memory" {
852+
eprintln!(
853+
"Warning: Agent '{}': 'safe-outputs: memory:' has moved to \
854+
'tools: cache-memory:'. Update your front matter to restore memory support.",
855+
front_matter.name
856+
);
857+
continue;
858+
}
838859
if !ALL_KNOWN_SAFE_OUTPUTS.contains(&key.as_str()) {
839860
eprintln!(
840861
"Warning: unrecognized safe-output tool '{}' — skipping (no registered tool matches this name)",
@@ -849,8 +870,8 @@ pub fn generate_enabled_tools_args(front_matter: &FrontMatter) -> String {
849870
}
850871

851872
if effective_mcp_tool_count == 0 {
852-
// Every user-specified key was either invalid, unrecognized, or non-MCP
853-
// (e.g. memory-only). Return empty to keep all tools available (backward compat).
873+
// Every user-specified key was either invalid or unrecognized.
874+
// Return empty to keep all tools available (backward compat).
854875
return String::new();
855876
}
856877

@@ -878,7 +899,7 @@ pub fn generate_enabled_tools_args(front_matter: &FrontMatter) -> String {
878899

879900
/// Validate that write-requiring safe-outputs have a write service connection configured.
880901
pub fn validate_write_permissions(front_matter: &FrontMatter) -> Result<()> {
881-
use crate::tools::WRITE_REQUIRING_SAFE_OUTPUTS;
902+
use crate::safeoutputs::WRITE_REQUIRING_SAFE_OUTPUTS;
882903

883904
let has_write_sc = front_matter
884905
.permissions
@@ -1170,6 +1191,8 @@ mod tests {
11701191
fm.tools = Some(crate::compile::types::ToolsConfig {
11711192
bash: Some(vec![":*".to_string()]),
11721193
edit: None,
1194+
cache_memory: None,
1195+
azure_devops: None,
11731196
});
11741197
let params = generate_copilot_params(&fm);
11751198
assert!(params.contains("--allow-tool \"shell(:*)\""));
@@ -1181,6 +1204,8 @@ mod tests {
11811204
fm.tools = Some(crate::compile::types::ToolsConfig {
11821205
bash: Some(vec![]),
11831206
edit: None,
1207+
cache_memory: None,
1208+
azure_devops: None,
11841209
});
11851210
let params = generate_copilot_params(&fm);
11861211
assert!(!params.contains("shell("));
@@ -1886,26 +1911,26 @@ mod tests {
18861911
}
18871912

18881913
#[test]
1889-
fn test_generate_enabled_tools_args_skips_memory_key() {
1890-
// `memory` is a non-MCP executor-only key. It must not appear in
1891-
// --enabled-tools or it would cause real MCP tools to be filtered out.
1914+
fn test_generate_enabled_tools_args_memory_no_longer_safe_output() {
1915+
// `memory` is no longer a safe-output key — it moved to `tools: cache-memory:`.
1916+
// If someone still puts it in safe-outputs, it should be treated as unrecognized
1917+
// and the real MCP tool should still be present.
18921918
let (fm, _) = parse_markdown(
1893-
"---\nname: test\ndescription: test\nsafe-outputs:\n memory:\n allowed-extensions:\n - .md\n create-pull-request:\n target-branch: main\n---\n"
1919+
"---\nname: test\ndescription: test\nsafe-outputs:\n create-pull-request:\n target-branch: main\n---\n"
18941920
).unwrap();
18951921
let args = generate_enabled_tools_args(&fm);
1896-
assert!(!args.contains("--enabled-tools memory"), "Non-MCP key 'memory' should be skipped");
18971922
assert!(args.contains("--enabled-tools create-pull-request"), "Real MCP tool should be present");
18981923
}
18991924

19001925
#[test]
1901-
fn test_generate_enabled_tools_args_memory_only_does_not_filter() {
1902-
// When `memory` is the only safe-output key, no --enabled-tools args should
1903-
// be generated so all tools remain available (backward compat).
1926+
fn test_generate_enabled_tools_args_empty_safe_outputs_no_filter() {
1927+
// When safe-outputs is empty, no --enabled-tools args should be generated
1928+
// so all tools remain available.
19041929
let (fm, _) = parse_markdown(
1905-
"---\nname: test\ndescription: test\nsafe-outputs:\n memory:\n allowed-extensions:\n - .md\n---\n"
1930+
"---\nname: test\ndescription: test\n---\n"
19061931
).unwrap();
19071932
let args = generate_enabled_tools_args(&fm);
1908-
assert!(args.is_empty(), "memory-only safe-outputs should produce no args (all tools available)");
1933+
assert!(args.is_empty(), "empty safe-outputs should produce no args (all tools available)");
19091934
}
19101935

19111936
// ─── parameter name validation ──────────────────────────────────────────

src/compile/onees.rs

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,11 @@ impl Compiler for OneESCompiler {
6262
let checkout_steps = generate_checkout_steps(&front_matter.checkout);
6363
let checkout_self = generate_checkout_self();
6464
let copilot_params = generate_copilot_params(front_matter);
65-
let has_memory = front_matter.safe_outputs.contains_key("memory");
65+
let has_memory = front_matter
66+
.tools
67+
.as_ref()
68+
.and_then(|t| t.cache_memory.as_ref())
69+
.is_some_and(|cm| cm.is_enabled());
6670
let parameters = build_parameters(&front_matter.parameters, has_memory);
6771
let parameters_yaml = generate_parameters(&parameters)?;
6872

0 commit comments

Comments
 (0)