refactor(compile): detect az at pipeline time so missing azure-cli no longer crashes 1ES

Reviewer-requested fix: static AWF bind-mounts for /opt/az and /usr/bin/az
would break `docker run` on runners without azure-cli pre-installed (notably
some 1ES self-hosted pools), failing the pipeline before the agent ever
started.

Replace the static mounts with a runtime detection prepare step that sets
the ADO pipeline variable AW_AZ_MOUNTS via `##vso[task.setvariable]` when
both /usr/bin/az and /opt/az exist on the host, or emits a `task.logissue`
warning and leaves the variable unset otherwise.

The AWF invocation in the compiled YAML now includes a single
`$(AW_AZ_MOUNTS) \` line in the --mount chain. ADO interpolates the variable
at step start: present -> the two --mount args appear; absent -> the line
collapses to whitespace. No new trait method is added; only the existing
`prepare_steps` hook is used.

- AzureCliExtension: required_awf_mounts() now returns []; prepare_steps()
  emits the detection bash step
- generate_awf_mounts: appends `$(AW_AZ_MOUNTS) \` when AzureCli is present
- Tests: rewrite static-mount assertions to assert the detection step + the
  pipeline variable injection, plus a regression guard that no static az
  mount is emitted
- Docs: docs/network.md and docs/tools.md updated with the runtime-detection
  design and operator implications

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: jamesadevine

docs/network.md

@@ -51,18 +51,54 @@ The following domains are always allowed. Most are defined in `CORE_ALLOWED_HOST
 
 ## Always-on Azure CLI (`az`)
 
-Every compiled pipeline mounts the host's `az` binary (from `/opt/az` and
-`/usr/bin/az`) into the AWF container and adds the Azure auth and
-management hosts listed above (`login.microsoftonline.com`,
-`login.windows.net`, `management.azure.com`, `graph.microsoft.com`,
-`aka.ms`) to the allowlist. This mirrors gh-aw's "assume `gh` is on the
-runner" model: agents can call `az` from their bash tool without
-opting in.
-
-The host is assumed to have `azure-cli` pre-installed. Microsoft-hosted
-`ubuntu-latest` agents satisfy this; 1ES self-hosted pool operators must
-bake `azure-cli` into their images. If `/opt/az` is missing on the host,
-the AWF mount will fail at runtime with a clear error.
+Every compiled pipeline adds the Azure auth and management hosts listed
+above (`login.microsoftonline.com`, `login.windows.net`,
+`management.azure.com`, `graph.microsoft.com`, `aka.ms`) to the AWF
+allowlist and emits a small *Detect Azure CLI on host* prepare step
+that runs early in the Agent job. This mirrors gh-aw's "assume `gh` is
+on the runner" model: agents can call `az` from their bash tool
+without opting in — *when the runner has it*.
+
+### Runtime detection and graceful degradation
+
+Because `azure-cli` is not universally pre-installed on every ADO
+runner image (notably some 1ES self-hosted pools), the compiler does
+**not** declare static AWF bind-mounts for `/opt/az` and `/usr/bin/az`.
+Static mounts would cause `docker run` to fail with "bind source path
+does not exist" on runners without `az`, breaking the pipeline before
+the agent ever started.
+
+Instead, the prepare step does the detection itself at pipeline time:
+
+* If both `/usr/bin/az` (the launcher shim) and `/opt/az` (the Python
+  venv that `az` actually runs in) exist on the host, the step sets
+  the ADO pipeline variable
+  `AW_AZ_MOUNTS=--mount /opt/az:/opt/az:ro --mount /usr/bin/az:/usr/bin/az:ro`
+  via `##vso[task.setvariable]`.
+* If either is missing, the step emits a
+  `##vso[task.logissue type=warning]` explaining `az` won't be
+  available inside the agent sandbox and leaves `AW_AZ_MOUNTS` unset
+  (which expands to the empty string).
+
+The AWF invocation in the compiled YAML then includes a literal
+`$(AW_AZ_MOUNTS) \` line on its own in the `--mount` chain.
+At step start, ADO interpolates that pipeline variable into the bash
+script: when az is present the two `--mount` args appear; when it's
+absent the line collapses to empty whitespace + the `\` continuation,
+which is a no-op.
+
+### Operator implications
+
+- **Microsoft-hosted `ubuntu-latest`**: `az` is detected, mounted, and
+  available inside the agent sandbox. Nothing to do.
+- **1ES self-hosted runners *with* azure-cli baked in**: same as above.
+- **1ES self-hosted runners *without* azure-cli**: the pipeline runs
+  successfully, but agents that invoke `az` get the standard
+  `command not found` inside the sandbox. The warning emitted by the
+  prepare step is visible in the ADO log as a yellow-flagged issue on
+  the build summary; treat it as a signal to either ignore (if no
+  agent on that runner needs `az`) or to install `azure-cli` on the
+  runner image.
 
 See [`docs/tools.md`](tools.md#built-in-clis) for the agent-facing
 contract (auth scope, available subcommands).

docs/tools.md

@@ -94,18 +94,37 @@ host is presumed to have each binary pre-installed.
 
 ### Azure CLI (`az`)
 
-Every compiled pipeline mounts the host's `az` binary into the AWF
-container (`/opt/az` + `/usr/bin/az`, read-only) and adds the Azure
-auth and management hosts (`login.microsoftonline.com`,
-`login.windows.net`, `management.azure.com`, `graph.microsoft.com`,
-`aka.ms`) to the AWF allowlist. The compiler does not install `az` —
-the host is assumed to already have `azure-cli` installed.
+Every compiled pipeline adds the Azure auth and management hosts
+(`login.microsoftonline.com`, `login.windows.net`,
+`management.azure.com`, `graph.microsoft.com`, `aka.ms`) to the AWF
+allowlist and emits a *Detect Azure CLI on host* prepare step in the
+Agent job. The compiler does not install `az`.
+
+**Runtime detection + graceful degradation.** The detection step does
+two things at pipeline time:
+
+1. If `/usr/bin/az` (the launcher shim) and `/opt/az` (the Python
+   venv that `az` runs in) both exist on the runner, it sets the
+   pipeline variable
+   `AW_AZ_MOUNTS=--mount /opt/az:/opt/az:ro --mount /usr/bin/az:/usr/bin/az:ro`.
+2. If either is missing, it emits a yellow ADO warning
+   (`##vso[task.logissue type=warning]`) and leaves the variable
+   unset.
+
+The AWF invocation includes a `$(AW_AZ_MOUNTS) \` line in its
+`--mount` chain. ADO expands the variable at step start: present →
+the two mounts appear; absent → the line collapses to nothing. No
+static `--mount` is emitted for `/opt/az` or `/usr/bin/az`, so the
+pipeline never crashes `docker run` with "bind source path does not
+exist" on runners without `az`. See
+[`docs/network.md`](network.md#always-on-azure-cli-az) for the full
+design.
 
 | Host posture                          | What you get                                              |
 | ------------------------------------- | --------------------------------------------------------- |
-| Microsoft-hosted `ubuntu-latest`      | Works out of the box (`az` is pre-installed)              |
-| 1ES self-hosted pool image            | Works if the pool operator baked `azure-cli` into the image |
-| Host missing `/opt/az`                | AWF mount fails at runtime with a clear error             |
+| Microsoft-hosted `ubuntu-latest`      | Detected → mounted → `az` available in the sandbox        |
+| 1ES self-hosted pool with `azure-cli` | Same as above                                             |
+| 1ES self-hosted pool *without* `az`   | Pipeline runs; warning in ADO log; `az` is `command not found` inside the sandbox |
 
 **Auth scope (important).** The compiler does not authenticate `az` for
 general use. Two paths are supported:

src/compile/common.rs

@@ -2909,15 +2909,33 @@ pub fn generate_awf_mounts(extensions: &[super::extensions::Extension]) -> Strin
         .flat_map(|ext| ext.required_awf_mounts())
         .collect();
 
-    if mounts.is_empty() {
+    // When the always-on AzureCli extension is enabled, append a
+    // pipeline-variable reference that expands at pipeline time to
+    // either `--mount /opt/az:/opt/az:ro --mount /usr/bin/az:/usr/bin/az:ro`
+    // (when the runner has azure-cli installed) or to nothing (when it
+    // doesn't). The detection + setvariable happens in
+    // `AzureCliExtension::prepare_steps`. This avoids static bind-mounts
+    // that would crash `docker run` on 1ES self-hosted runners without
+    // azure-cli pre-installed.
+    let inject_az_var = extensions
+        .iter()
+        .any(|ext| matches!(ext, super::extensions::Extension::AzureCli(_)));
+
+    if mounts.is_empty() && !inject_az_var {
         return "\\".to_string();
     }
 
-    mounts
+    let mut lines: Vec<String> = mounts
         .iter()
         .map(|m| format!("--mount \"{}\" \\", m))
-        .collect::<Vec<_>>()
-        .join("\n")
+        .collect();
+    if inject_az_var {
+        // Unquoted on purpose: bash word-splits the pipeline-var value
+        // into separate `--mount <spec>` tokens. The value contains only
+        // path chars + `:` + spaces, no shell metachars.
+        lines.push("$(AW_AZ_MOUNTS) \\".to_string());
+    }
+    lines.join("\n")
 }
 
 /// Generates a dedicated pipeline step that writes a `GITHUB_PATH` file
@@ -6779,20 +6797,24 @@ safe-outputs:
     #[test]
     fn test_generate_awf_mounts_no_extensions() {
         // Even with a minimal front matter, the always-on Azure CLI
-        // extension contributes its two AWF mounts (/opt/az + /usr/bin/az).
-        // The "no mounts" name is historical; this test now verifies the
-        // always-on baseline.
+        // extension contributes a `$(AW_AZ_MOUNTS) \` injection line
+        // (no static mounts — those are runtime-detected by the
+        // AzureCli prepare step which sets the pipeline variable).
+        // The "no mounts" name is historical; this test now verifies
+        // the always-on baseline.
         let fm = minimal_front_matter();
         let exts = crate::compile::extensions::collect_extensions(&fm);
         let _ctx = crate::compile::extensions::CompileContext::for_test(&fm);
         let result = generate_awf_mounts(&exts);
         assert!(
-            result.contains(r#"--mount "/opt/az:/opt/az:ro""#),
-            "always-on Azure CLI mount /opt/az should be present: {result}"
+            result.contains("$(AW_AZ_MOUNTS) \\"),
+            "always-on Azure CLI injection line $(AW_AZ_MOUNTS) \\ should be present \
+             (so the AzureCli prepare step's pipeline variable expands into runtime mounts): {result}"
         );
         assert!(
-            result.contains(r#"--mount "/usr/bin/az:/usr/bin/az:ro""#),
-            "always-on Azure CLI mount /usr/bin/az should be present: {result}"
+            !result.contains(r#"--mount "/opt/az:/opt/az:ro""#),
+            "must NOT emit a static /opt/az --mount — that would crash docker run on \
+             runners without azure-cli. The mount is contributed via $(AW_AZ_MOUNTS) instead: {result}"
         );
         assert!(
             result.ends_with(" \\"),