Skip to content

Commit 991621a

Browse files
feat(runtimes): add unified Node.js and Python runtime extensions (#400)
* feat(runtimes): add unified Node.js and Python runtime extensions Add Python and Node.js runtimes with consistent architecture matching the existing Lean runtime pattern: - Python: UsePythonVersion@0, PipAuthenticate@1, PIP_INDEX_URL/UV_DEFAULT_INDEX env vars - Node.js: NodeTool@0 (inline, decoupled from ado-script), npmAuthenticate@0, NPM_CONFIG_REGISTRY env var - Both use flat feed-url: field with env var injection via agent_env_vars() - Both accept config: field (recognized but errors if used, reserved for AWF proxy-auth) - Shared validate_feed_url() in validate.rs for injection checks - agent_env_vars() trait method on CompilerExtension with BLOCKED_ENV_KEYS validation - No AWF mounts/PATH prepends needed (hostedtoolcache auto-mounted by AWF) Unifies the approaches from PRs #398 and #399 into a single consistent implementation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(runtimes): ensure .npmrc exists before npmAuthenticate@0 npmAuthenticate@0 requires workingFile to point at an existing file, unlike PipAuthenticate@1. Emit a bash step that creates a minimal .npmrc (with the configured registry or default npmjs) when one does not already exist, preserving any repo-checked-in .npmrc. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(runtimes): fix dead-code validation ordering, accept node config with warning - Python: swap mutual-exclusivity check before not-yet-supported error so both paths are reachable - Node: accept config: with a warning that .npmrc won't be available inside AWF yet (instead of hard error), check mutual exclusivity first - Add tests for mutual-exclusivity errors on both runtimes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(runtimes): reject double-quote in feed URLs and agent env var values A double-quote in a feed-url or extension env var value would produce malformed YAML in the generated pipeline (the value is emitted as KEY: "value"). Reject at validation time in both validate_feed_url() and collect_agent_env_vars() with clear error messages. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(runtimes): Python config: produces warning instead of error Align with Node.js behavior — accept config: with a warning that the config file will not be available inside the AWF agent environment yet, rather than a hard compile error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor(runtimes): make auth tasks conditional on feed-url or config PipAuthenticate@1 and npmAuthenticate@0 (plus ensure-npmrc) are now only emitted when feed-url: or config: is set. Users who enable runtimes: python: true or runtimes: node: true without an internal feed no longer get unnecessary auth steps in their pipeline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(runtimes): reject single-quote in feed URLs, fix doc comments, soften uv prompt Three fixes: 1. validate_feed_url() now rejects single-quote characters alongside double-quotes — a single quote in the feed URL would break the bash single-quoted string in generate_ensure_npmrc's echo command. 2. Doc comments on generate_pip_authenticate() and generate_npm_authenticate() corrected to say "emitted when feed-url or config is set" instead of "emitted unconditionally", matching the actual conditional behavior in prepare_steps(). 3. Python prompt supplement no longer claims uv is "pre-installed" — ADO hosted runners don't ship uv. Now says "install it first with pip install uv" to avoid command-not-found errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(runtimes): harden env var collection, fix docs and bash quoting Security & correctness fixes: 1. collect_agent_env_vars now calls reject_pipeline_injection (covers ADO expressions, pipeline commands, template markers, newlines) instead of only contains_pipeline_command. Also rejects single quotes alongside double quotes. 2. collect_agent_env_vars now deduplicates env var keys — bails on collision instead of silently emitting duplicate YAML keys. 3. generate_ensure_npmrc diagnostic echo lines switched from double-quotes to single-quotes, preventing ${VAR} shell expansion if a feed URL contained that pattern. 4. docs/runtimes.md corrected: auth tasks are conditional on feed-url or config being set, not unconditional. Config field descriptions updated to reflect warning-not-error behavior. Added note about PipAuthenticate@1 empty artifactFeeds limitation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor(runtimes): PipAuthenticate requires feed-url, not config alone PipAuthenticate@1 with empty artifactFeeds doesn't authenticate to any specific feed. Only emit it when feed-url is set — config alone is not sufficient since the config file won't be available in AWF yet anyway. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent da83c03 commit 991621a

14 files changed

Lines changed: 1100 additions & 5 deletions

File tree

AGENTS.md

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -99,8 +99,14 @@ Every compiled pipeline runs as three sequential jobs:
9999
│ │ └── upload_workitem_attachment.rs
100100
│ ├── runtimes/ # Runtime environment implementations (one dir per runtime)
101101
│ │ ├── mod.rs # Module entry point
102-
│ │ └── lean/ # Lean 4 theorem prover runtime
103-
│ │ ├── mod.rs # Config types, install helpers
102+
│ │ ├── lean/ # Lean 4 theorem prover runtime
103+
│ │ │ ├── mod.rs # Config types, install helpers
104+
│ │ │ └── extension.rs # CompilerExtension impl
105+
│ │ ├── python/ # Python runtime
106+
│ │ │ ├── mod.rs # Config types, install/auth helpers
107+
│ │ │ └── extension.rs # CompilerExtension impl
108+
│ │ └── node/ # Node.js runtime
109+
│ │ ├── mod.rs # Config types, install/auth helpers
104110
│ │ └── extension.rs # CompilerExtension impl
105111
│ ├── data/
106112
│ │ ├── base.yml # Base pipeline template for standalone
@@ -156,8 +162,8 @@ index to jump to the right page.
156162
in the pipeline UI, including the auto-injected `clearMemory` parameter.
157163
- [`docs/tools.md`](docs/tools.md)`tools:` configuration (bash allow-list,
158164
`edit`, `cache-memory`, `azure-devops` MCP).
159-
- [`docs/runtimes.md`](docs/runtimes.md)`runtimes:` configuration (currently
160-
Lean 4).
165+
- [`docs/runtimes.md`](docs/runtimes.md)`runtimes:` configuration (Lean 4,
166+
Python, Node.js).
161167
- [`docs/targets.md`](docs/targets.md) — target platforms: `standalone` and
162168
`1es`.
163169
- [`docs/safe-outputs.md`](docs/safe-outputs.md) — full reference for every

docs/runtimes.md

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,3 +33,87 @@ When enabled, the compiler:
3333
- Emits a compile-time warning if `tools.bash` is empty (Lean requires bash access)
3434

3535
**Note:** In the 1ES target, the bash command allow-list is updated but elan installation must be done manually via `steps:` front matter. The 1ES target handles network isolation separately.
36+
37+
### Python (`python:`)
38+
39+
Python runtime. Auto-installs Python via `UsePythonVersion@0`, emits `PipAuthenticate@1` for internal feed access, adds Python ecosystem domains to the AWF network allowlist, extends the bash command allow-list, and optionally injects feed URL env vars for pip and uv.
40+
41+
```yaml
42+
# Simple enablement (installs default Python 3.x)
43+
runtimes:
44+
python: true
45+
46+
# With options (pin version, configure feed)
47+
runtimes:
48+
python:
49+
version: "3.12"
50+
feed-url: "https://pkgs.dev.azure.com/myorg/_packaging/myfeed/pypi/simple/"
51+
```
52+
53+
**Fields:**
54+
55+
| Field | Type | Description |
56+
|-------|------|-------------|
57+
| `version` | string | Python version to install (e.g., `"3.12"`, `"3.11"`). Passed to `UsePythonVersion@0` `versionSpec`. Defaults to latest 3.x. |
58+
| `feed-url` | string | Internal PyPI feed URL. Injects `PIP_INDEX_URL` and `UV_DEFAULT_INDEX` env vars into the agent environment. |
59+
| `config` | string | Path to a pip/uv config file. Accepted with a warning — the file will not be available inside the AWF agent environment until proxy-auth support lands. |
60+
61+
When enabled, the compiler:
62+
- Injects `UsePythonVersion@0` into `{{ prepare_steps }}` (runs before AWF)
63+
- If `feed-url` is set, also injects `PipAuthenticate@1` to authenticate the ADO build service identity for internal feeds
64+
- Auto-adds `python`, `python3`, `pip`, `pip3`, `uv` to the bash command allow-list
65+
- Adds Python ecosystem domains to the network allowlist (pypi.org, pythonhosted.org, etc.)
66+
- If `feed-url` is set, injects `PIP_INDEX_URL` and `UV_DEFAULT_INDEX` env vars into the agent environment
67+
- Appends a prompt supplement informing the agent about Python availability
68+
- No AWF mounts or PATH prepends needed — `UsePythonVersion@0` installs to `/opt/hostedtoolcache` (auto-mounted by AWF) and publishes PATH entries that AWF merges via `$GITHUB_PATH`
69+
70+
**Note:** `PipAuthenticate@1` is currently emitted with an empty `artifactFeeds` input, which configures credentials for all feeds accessible to the build service identity. If your internal feed requires scoped authentication to a specific Azure Artifacts feed, this may need future refinement.
71+
72+
### Node.js (`node:`)
73+
74+
Node.js runtime. Auto-installs Node.js via `NodeTool@0`, emits `npmAuthenticate@0` for internal feed access, adds Node ecosystem domains to the AWF network allowlist, extends the bash command allow-list, and optionally injects feed URL env vars for npm.
75+
76+
```yaml
77+
# Simple enablement (installs default Node LTS)
78+
runtimes:
79+
node: true
80+
81+
# With options (pin version, configure feed)
82+
runtimes:
83+
node:
84+
version: "22.x"
85+
feed-url: "https://pkgs.dev.azure.com/ORG/PROJECT/_packaging/FEED/npm/registry/"
86+
```
87+
88+
**Fields:**
89+
90+
| Field | Type | Description |
91+
|-------|------|-------------|
92+
| `version` | string | Node.js version to install (e.g., `"22.x"`, `"20.x"`). Passed to `NodeTool@0` `versionSpec`. Defaults to `"22.x"`. |
93+
| `feed-url` | string | Internal npm registry URL. Injects `NPM_CONFIG_REGISTRY` env var into the agent environment. |
94+
| `config` | string | Path to an .npmrc config file. Accepted with a warning — the file will not be available inside the AWF agent environment until proxy-auth support lands. |
95+
96+
When enabled, the compiler:
97+
- Injects `NodeTool@0` into `{{ prepare_steps }}` (runs before AWF)
98+
- If `feed-url` or `config` is set, also injects `npmAuthenticate@0` (and an ensure-`.npmrc` step) to authenticate the ADO build service identity for internal feeds
99+
- Auto-adds `node`, `npm`, `npx` to the bash command allow-list
100+
- Adds Node ecosystem domains to the network allowlist (npmjs.org, nodejs.org, etc.)
101+
- If `feed-url` is set, injects `NPM_CONFIG_REGISTRY` env var into the agent environment
102+
- Appends a prompt supplement informing the agent about Node.js availability
103+
- No AWF mounts or PATH prepends needed — `NodeTool@0` installs to `/opt/hostedtoolcache` (auto-mounted by AWF) and publishes PATH entries that AWF merges via `$GITHUB_PATH`
104+
- Note: AWF overlays `~/.npmrc` with `/dev/null` for credential security — the `NPM_CONFIG_REGISTRY` env var approach avoids conflicting with this overlay
105+
106+
### Combining Runtimes
107+
108+
Multiple runtimes can be enabled simultaneously:
109+
110+
```yaml
111+
runtimes:
112+
python:
113+
version: "3.12"
114+
node:
115+
version: "22.x"
116+
lean: true
117+
```
118+
119+
All runtime extensions are sorted into `ExtensionPhase::Runtime` and execute before tool extensions (`ExtensionPhase::Tool`), ensuring language toolchains are available before any tools that depend on them.

src/compile/common.rs

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1912,6 +1912,71 @@ pub fn collect_awf_path_prepends(extensions: &[super::extensions::Extension]) ->
19121912
.collect()
19131913
}
19141914

1915+
/// Collects `agent_env_vars()` from all extensions, validates keys against
1916+
/// `BLOCKED_ENV_KEYS`, deduplicates (bails on collision), and formats them
1917+
/// as YAML `KEY: "value"` lines for injection into the `{{ engine_env }}` block.
1918+
///
1919+
/// Returns an empty string if no extensions declare env vars.
1920+
pub fn collect_agent_env_vars(extensions: &[super::extensions::Extension]) -> anyhow::Result<String> {
1921+
use crate::engine::BLOCKED_ENV_KEYS;
1922+
use crate::validate;
1923+
use std::collections::HashSet;
1924+
1925+
let mut lines = Vec::new();
1926+
let mut seen_keys = HashSet::new();
1927+
1928+
for ext in extensions {
1929+
for (key, value) in ext.agent_env_vars() {
1930+
// Deduplicate: bail on collision
1931+
if !seen_keys.insert(key.clone()) {
1932+
anyhow::bail!(
1933+
"Extension '{}' declares agent env var '{}' which was already declared \
1934+
by a previous extension. Each env var key must be unique.",
1935+
ext.name(),
1936+
key,
1937+
);
1938+
}
1939+
1940+
// Validate key is not blocked
1941+
if BLOCKED_ENV_KEYS.iter().any(|blocked| key.eq_ignore_ascii_case(blocked)) {
1942+
anyhow::bail!(
1943+
"Extension '{}' declares agent env var '{}' which conflicts with a \
1944+
compiler-controlled environment variable.",
1945+
ext.name(),
1946+
key,
1947+
);
1948+
}
1949+
1950+
// Validate key format
1951+
if !validate::is_valid_env_var_name(&key) {
1952+
anyhow::bail!(
1953+
"Extension '{}' declares agent env var '{}' with invalid key format. \
1954+
Keys must contain only ASCII alphanumerics and underscores.",
1955+
ext.name(),
1956+
key,
1957+
);
1958+
}
1959+
1960+
// Validate value for injection (defence in depth — covers ADO expressions,
1961+
// pipeline commands, template markers, and newlines)
1962+
validate::reject_pipeline_injection(&value, &format!("agent env var '{key}'"))?;
1963+
1964+
if value.contains('"') || value.contains('\'') {
1965+
anyhow::bail!(
1966+
"Extension '{}' agent env var '{}' value contains a quote character \
1967+
which would produce malformed YAML or bash syntax.",
1968+
ext.name(),
1969+
key,
1970+
);
1971+
}
1972+
1973+
lines.push(format!("{key}: \"{value}\""));
1974+
}
1975+
}
1976+
1977+
Ok(lines.join("\n"))
1978+
}
1979+
19151980
// ==================== Shared compile flow ====================
19161981

19171982
/// Target-specific overrides for the shared compile flow.
@@ -2104,6 +2169,12 @@ pub async fn compile_shared(
21042169
if !awf_path_env.is_empty() {
21052170
engine_env = format!("{engine_env}\n{awf_path_env}");
21062171
}
2172+
2173+
// Append extension-declared agent env vars (e.g., PIP_INDEX_URL, NPM_CONFIG_REGISTRY)
2174+
let agent_env = collect_agent_env_vars(extensions)?;
2175+
if !agent_env.is_empty() {
2176+
engine_env = format!("{engine_env}\n{agent_env}");
2177+
}
21072178
let engine_log_dir = ctx.engine.log_dir();
21082179
let acquire_write_token = generate_acquire_ado_token(
21092180
front_matter
@@ -2505,6 +2576,8 @@ mod tests {
25052576
});
25062577
fm.runtimes = Some(crate::compile::types::RuntimesConfig {
25072578
lean: Some(crate::runtimes::lean::LeanRuntimeConfig::Enabled(true)),
2579+
python: None,
2580+
node: None,
25082581
});
25092582
let params = CompileContext::for_test(&fm).engine.args(&fm, &crate::compile::extensions::collect_extensions(&fm)).unwrap();
25102583
assert!(params.contains("shell(lean)"), "lean command should be allowed");
@@ -2525,6 +2598,8 @@ mod tests {
25252598
});
25262599
fm.runtimes = Some(crate::compile::types::RuntimesConfig {
25272600
lean: Some(crate::runtimes::lean::LeanRuntimeConfig::Enabled(true)),
2601+
python: None,
2602+
node: None,
25282603
});
25292604
let params = CompileContext::for_test(&fm).engine.args(&fm, &crate::compile::extensions::collect_extensions(&fm)).unwrap();
25302605
assert!(params.contains("--allow-all-tools"), "wildcard should use --allow-all-tools");

src/compile/extensions/mod.rs

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -324,6 +324,17 @@ pub trait CompilerExtension {
324324
fn awf_path_prepends(&self) -> Vec<String> {
325325
vec![]
326326
}
327+
328+
/// Environment variables to inject into the agent execution environment.
329+
///
330+
/// Returns `(key, value)` pairs that are emitted as `KEY: "value"` in
331+
/// the `{{ engine_env }}` YAML block. Used by runtimes to configure
332+
/// package managers via env vars (e.g., `PIP_INDEX_URL`, `NPM_CONFIG_REGISTRY`).
333+
///
334+
/// Keys are validated against `BLOCKED_ENV_KEYS` at collection time.
335+
fn agent_env_vars(&self) -> Vec<(String, String)> {
336+
vec![]
337+
}
327338
}
328339

329340
/// Mount access mode for an AWF bind mount.
@@ -534,6 +545,9 @@ macro_rules! extension_enum {
534545
fn awf_path_prepends(&self) -> Vec<String> {
535546
match self { $( $Enum::$Variant(e) => e.awf_path_prepends(), )+ }
536547
}
548+
fn agent_env_vars(&self) -> Vec<(String, String)> {
549+
match self { $( $Enum::$Variant(e) => e.agent_env_vars(), )+ }
550+
}
537551
}
538552
};
539553
}
@@ -547,6 +561,8 @@ pub use crate::tools::azure_devops::AzureDevOpsExtension;
547561
pub use crate::tools::cache_memory::CacheMemoryExtension;
548562
pub use github::GitHubExtension;
549563
pub use crate::runtimes::lean::LeanExtension;
564+
pub use crate::runtimes::node::NodeExtension;
565+
pub use crate::runtimes::python::PythonExtension;
550566
pub use safe_outputs::SafeOutputsExtension;
551567
pub use trigger_filters::TriggerFiltersExtension;
552568

@@ -559,6 +575,8 @@ extension_enum! {
559575
GitHub(GitHubExtension),
560576
SafeOutputs(SafeOutputsExtension),
561577
Lean(LeanExtension),
578+
Python(PythonExtension),
579+
Node(NodeExtension),
562580
AzureDevOps(AzureDevOpsExtension),
563581
CacheMemory(CacheMemoryExtension),
564582
TriggerFilters(TriggerFiltersExtension),
@@ -593,6 +611,16 @@ pub fn collect_extensions(front_matter: &FrontMatter) -> Vec<Extension> {
593611
extensions.push(Extension::Lean(LeanExtension::new(lean.clone())));
594612
}
595613
}
614+
if let Some(python) = front_matter.runtimes.as_ref().and_then(|r| r.python.as_ref()) {
615+
if python.is_enabled() {
616+
extensions.push(Extension::Python(PythonExtension::new(python.clone())));
617+
}
618+
}
619+
if let Some(node) = front_matter.runtimes.as_ref().and_then(|r| r.node.as_ref()) {
620+
if node.is_enabled() {
621+
extensions.push(Extension::Node(NodeExtension::new(node.clone())));
622+
}
623+
}
596624

597625
// ── First-party tools (ExtensionPhase::Tool) ──
598626
if let Some(tools) = front_matter.tools.as_ref() {

0 commit comments

Comments
 (0)