feat(smoke): run agentic smoke against compiler candidates (#1628)

* feat(smoke): run agentic smoke against compiler candidates

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: a7017732-e510-4ed3-84fb-47ce7120230e

* refactor(compile): address pipeline artifact review feedback

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: a7017732-e510-4ed3-84fb-47ce7120230e

* fix(smoke): align candidate integrity metadata

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: a7017732-e510-4ed3-84fb-47ce7120230e

* chore(smoke): add AgentPlayground secret rotation script

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: a7017732-e510-4ed3-84fb-47ce7120230e

* fix(smoke): keep release pipelines off pull requests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: a7017732-e510-4ed3-84fb-47ce7120230e

Author: jamesadevine

AGENTS.md

@@ -342,10 +342,11 @@ index to jump to the right page.
   issues from dogfood pipelines). NOT a regular safe-output.
 - [`docs/supply-chain.md`](docs/supply-chain.md) — optional `supply-chain:`
   front-matter section that mirrors the compiler, AWF binary, ado-script
-  bundle, and AWF/MCPG images from an internal Azure DevOps Artifacts feed
-  and/or container registry (NuGet `DownloadPackage@1` + ACR `az acr login`),
-  with asymmetric auth (feed defaults to `$(System.AccessToken)`; registry
-  requires a service connection).
+  bundle, and AWF/MCPG images from an internal Azure DevOps Artifacts feed,
+  an exact provenance-checked pipeline artifact, and/or a container registry.
+  Feed and pipeline artifact are mutually exclusive; registry is independent.
+  Feed defaults to `$(System.AccessToken)`, while registry requires a service
+  connection and the pipeline artifact uses the build identity.
 
 ### Compiler internals & operations
 
@@ -442,8 +443,8 @@ Following the gh-aw security model:
    `src/compile/agentic_pipeline.rs::run_agent_step`). AWF runs rootlessly in
    strict topology mode; the Agent reaches trusted MCPG through
    `--topology-attach awmg-mcpg`, not through host access. All other ADO steps —
-   binary/bundle downloads, `docker pull`, ACR/NuGet auth (including the
-   `supply-chain:` mirror fetches) — run *outside* the sandbox with the build
+   binary/bundle downloads, `docker pull`, ACR/NuGet auth (including feed and
+   pipeline-artifact `supply-chain:` fetches) — run *outside* the sandbox with the build
    agent pool's normal network, so they do **not** need entries in the AWF
    allowlist. Air-gapping the build agent itself from GitHub/GHCR is the agent
    pool's network policy, not AWF.
@@ -496,6 +497,17 @@ anything it flags. If a finding is genuinely intentional, add a
 `# shellcheck disable=SCxxxx` comment immediately above the offending line in
 the bash body — shellcheck honours the directive and it's inert at runtime.
 
+### Release-owned smoke lock files
+
+`tests/safe-outputs/*.lock.yml` are the latest-release customer contract. Do
+not regenerate them with `cargo run -- compile` from an unreleased checkout:
+their runtime integrity step downloads the released compiler, so development
+output can drift even while Cargo still reports the same semver. Compiler PRs
+and nightly `main` are exercised by `tests/compiler-smoke-e2e/`, which
+recompiles all five workflows in a temporary worktree and stages them on an
+ephemeral `ado-aw-mirror` ref. The release workflow updates the checked-in
+locks only after matching release assets exist.
+
 ## Common Tasks
 
 ### Compile a markdown pipeline

README.md

@@ -729,7 +729,8 @@ index to jump to the right page.
 - [`docs/safe-output-permissions.md`](docs/safe-output-permissions.md) —
   diagnosis and fix reference for Stage 3 permission failures (401/403).
 - [`docs/supply-chain.md`](docs/supply-chain.md) — optional `supply-chain:`
-  front-matter section for internal feed/registry mirrors.
+  front-matter section for internal feed/registry mirrors and exact
+  provenance-checked pipeline artifacts.
 - [`docs/ado-aw-debug.md`](docs/ado-aw-debug.md) — debug-only `ado-aw-debug:`
   front-matter section (`skip-integrity`, `create-issue`).
 

docs/front-matter.md

@@ -217,10 +217,15 @@ supply-chain:                  # optional internal supply-chain mirror (see docs
   feed:                          # mirror binaries (compiler, AWF, ado-script) from an ADO Artifacts feed
     name: my-project/my-feed     # feed name or project/feed; scalar `feed: my-feed` shorthand also works
     service-connection: feed-conn  # optional; omit for same-org feeds (uses $(System.AccessToken))
+  # pipeline-artifact:           # alternative binary source; mutually exclusive with feed
+  #   project: AgentPlayground   # validated ADO project name or GUID
+  #   definition-id: 2560        # positive producer pipeline definition ID
+  #   run-id: 630001             # positive, exact producer build ID
+  #   artifact: ado-aw-candidate # complete payload/checksum/provenance artifact
   registry:                      # mirror AWF/MCPG images from an internal ACR
     name: myacr.azurecr.io/mirror  # registry host or base path (artifact names kept under it)
     service-connection: acr-conn   # REQUIRED when registry is set (ACR has no System.AccessToken path)
-  service-connection: shared-conn  # optional shared fallback for whichever target omits its own
+  service-connection: shared-conn  # optional feed/registry fallback; never applies to pipeline-artifact
 # ado-aw-debug:                 # debug-only knobs; see docs/ado-aw-debug.md
 #   skip-integrity: false       # omit generated pipeline integrity verification
 #   create-issue: false         # dogfood-only GitHub issue filing for debug reports

docs/supply-chain.md

@@ -11,10 +11,10 @@ GitHub / GHCR at run time:
 | 4 | AWF + MCPG container images | `ghcr.io/github/...` |
 
 The optional `supply-chain:` front-matter section reroutes these fetches to an
-**internal Azure DevOps Artifacts feed** (for the binaries #1–#3) and/or an
-**internal container registry** (for the images #4). This is intended for
-supply-chain-hardened environments where the build agent pool cannot reach
-GitHub / GHCR.
+**internal Azure DevOps Artifacts feed** or one exact **Azure DevOps pipeline
+artifact** (for the binaries #1–#3), and/or an **internal container registry**
+(for the images #4). This is intended for supply-chain-hardened environments
+where the build agent pool cannot reach GitHub / GHCR.
 
 When `supply-chain:` is omitted, the generated pipeline is byte-for-byte
 identical to before — there is no behavioural change for existing agents.
@@ -26,6 +26,12 @@ supply-chain:
   feed:                          # mirrors binaries #1, #2, #3
     name: my-project/my-feed     # feed name or "project/feed"
     service-connection: feed-conn  # optional (see Authentication)
+  # Alternative to feed (the two are mutually exclusive):
+  # pipeline-artifact:
+  #   project: AgentPlayground
+  #   definition-id: 2560
+  #   run-id: 630001
+  #   artifact: ado-aw-candidate
   registry:                      # mirrors images #4
     name: myacr.azurecr.io/mirror  # registry host or base path
     service-connection: acr-conn   # required when registry is set
@@ -35,10 +41,12 @@ supply-chain:
 | Field | Type | Required | Purpose |
 |-------|------|----------|---------|
 | `feed` | scalar **or** `{ name, service-connection }` | optional | Enables the binary mirror (#1–#3). A bare string is shorthand for `{ name: <string> }`. |
-| `registry` | scalar **or** `{ name, service-connection }` | optional | Enables the image mirror (#4). |
-| `service-connection` | string | optional | Shared fallback connection used by whichever target does not declare its own. |
+| `pipeline-artifact` | `{ project, definition-id, run-id, artifact }` | optional | Uses one exact producer run as the complete binary source (#1–#3). Mutually exclusive with `feed`. |
+| `registry` | scalar **or** `{ name, service-connection }` | optional | Enables the image mirror (#4), independently of either binary source. |
+| `service-connection` | string | optional | Shared fallback connection for `feed` and `registry`; it does not apply to `pipeline-artifact`. |
 
-`feed` and `registry` are **independent** — set either, both, or neither.
+`feed` and `pipeline-artifact` are mutually exclusive. `registry` is
+independent and may accompany either binary source.
 
 ### Scalar shorthand
 
@@ -71,6 +79,28 @@ role and `NuGetAuthenticate@1` authenticates automatically via
 `$(System.AccessToken)`. Set a `service-connection` only for cross-org or
 external feeds.
 
+### Pipeline artifact (binaries)
+
+The pipeline-artifact source is immutable configuration: the generated
+`DownloadPipelineArtifact@2` tasks always use `source: specific`,
+`runVersion: specific`, the configured project, definition ID (`pipeline`),
+run ID (`runId`), and artifact name. The compiler never selects latest runs,
+tags, or triggering pipelines.
+
+`project` must be an Azure DevOps project name or canonical GUID;
+`definition-id` and `run-id` must be positive integers; and `artifact` follows
+Azure DevOps artifact-name rules. The current build identity must have
+permission to read the configured project, pipeline run, and artifact.
+The producer must be in the same Azure DevOps organization; cross-organization
+pipeline-artifact downloads are not supported by this source.
+`supply-chain.service-connection` is not used for this source, and no
+`NuGetAuthenticate@1` step is emitted.
+
+With an exact run ID, the artifact is consumable as soon as its publish task
+has completed even if later jobs keep the producer run active. Consumers
+queued before the named artifact exists fail; producers should confirm artifact
+visibility before queueing them.
+
 ### Registry (images)
 
 The image mirror authenticates with `az acr login` (`AzureCLI@2`) using the
@@ -96,7 +126,7 @@ used by `permissions:`), passed to `AzureCLI@2` as `azureSubscription`.
 > `registry.name` to the canonical `*.azurecr.io` login server so the derived
 > registry name is correct.
 
-## What the feed and registry must contain
+## What the binary source and registry must contain
 
 Versions stay **pinned by the generating compiler** — the internal mirror must
 host the exact pinned versions. A mirror that is a few days behind is fine: use
@@ -119,6 +149,47 @@ payload, so the package simply needs to carry the same files (and matching
 `sha256sum -c checksums.txt` is preserved**, so the mirror must ship the
 matching `checksums.txt`.
 
+### Pipeline artifact
+
+The single configured artifact is a complete source and must contain exactly
+one file with each of these names (directory nesting is allowed):
+
+- `ado-aw-linux-x64`
+- `awf-linux-x64`
+- `ado-script.zip`
+- `checksums.txt`
+- `provenance.json`
+
+`checksums.txt` must have exactly one standard sha256sum entry for each payload,
+with the exact filename. Each consumer locates its payload plus the shared
+manifest and provenance document, verifies the exact checksum entry, then
+preserves the normal chmod/move/unzip/PATH behavior.
+
+`provenance.json` must be a JSON object using this Phase 1 contract:
+
+```json
+{
+  "schema": "ado-aw/candidate-artifact/1",
+  "producer_definition_id": 2560,
+  "producer_build_id": 630001,
+  "repository": "githubnext/ado-aw",
+  "source_ref": "refs/heads/main",
+  "source_version": "0123456789abcdef0123456789abcdef01234567",
+  "reason": "IndividualCI",
+  "compiler_version": "0.45.1",
+  "awf_version": "0.27.32"
+}
+```
+
+The required identity fields are exactly `schema`, `producer_definition_id`,
+and `producer_build_id`; both producer IDs are JSON numbers. Compilation embeds
+the configured IDs, and every consumer fails closed if the schema or either
+producer ID is missing, non-numeric, or mismatched. `repository`, `source_ref`,
+`source_version`, `reason`, `compiler_version`, and `awf_version` are optional
+diagnostic fields: when present, the generated step includes them in the
+validated, non-secret provenance output. Their absence does not invalidate an
+otherwise valid artifact.
+
 ### Registry images
 
 `registry.name` is a registry **host or base path** — teams generally cannot
@@ -175,6 +246,31 @@ supply-chain:
   feed: my-internal-feed
 ```
 
+All binaries from one exact candidate pipeline run:
+
+```yaml
+supply-chain:
+  pipeline-artifact:
+    project: AgentPlayground
+    definition-id: 2560
+    run-id: 630001
+    artifact: ado-aw-candidate
+```
+
+Candidate binaries plus mirrored images:
+
+```yaml
+supply-chain:
+  pipeline-artifact:
+    project: AgentPlayground
+    definition-id: 2560
+    run-id: 630001
+    artifact: ado-aw-candidate
+  registry:
+    name: myacr.azurecr.io
+    service-connection: acr-conn
+```
+
 Images only:
 
 ```yaml
@@ -186,9 +282,10 @@ supply-chain:
 
 ## Network isolation note
 
-The mirror fetches (`NuGetAuthenticate@1`, `DownloadPackage@1`, `docker pull`,
-`az acr login`) run as ordinary ADO steps on the build agent — **outside** the
-AWF network-isolation sandbox, which wraps only the copilot agent command.
+The mirror fetches (`NuGetAuthenticate@1`, `DownloadPackage@1`,
+`DownloadPipelineArtifact@2`, `docker pull`, `az acr login`) run as ordinary
+ADO steps on the build agent — **outside** the AWF network-isolation sandbox,
+which wraps only the copilot agent command.
 Consequently:
 
 - The feed/registry hosts are **not** added to the agent's AWF

rotate-agentplayground-secrets.ps1

@@ -0,0 +1,98 @@
+[CmdletBinding()]
+param(
+    [string] $Organization = "msazuresphere",
+    [string] $Project = "AgentPlayground"
+)
+
+Set-StrictMode -Version Latest
+$ErrorActionPreference = "Stop"
+
+$copilotDefinitionIds = @(2545, 2546, 2547, 2548, 2549, 2554, 2555, 2556, 2557, 2558)
+$reporterDefinitionIds = @(2549, 2558)
+$executorDefinitionIds = @(2550)
+$triggerDefinitionIds = @(2551)
+$allDefinitionIds = @(
+    $copilotDefinitionIds
+    $executorDefinitionIds
+    $triggerDefinitionIds
+) | Sort-Object -Unique
+
+function Set-AdoAwSecret {
+    param(
+        [Parameter(Mandatory)]
+        [string] $Name,
+
+        [Parameter(Mandatory)]
+        [string] $Value,
+
+        [Parameter(Mandatory)]
+        [int[]] $DefinitionIds
+    )
+
+    Write-Host "Setting $Name on definitions $($DefinitionIds -join ',')..."
+    $Value | & cargo run --quiet -- secrets set $Name --value-stdin `
+        --org $Organization `
+        --project $Project `
+        --definition-ids ($DefinitionIds -join ",")
+
+    if ($LASTEXITCODE -ne 0) {
+        throw "Failed to set $Name on definitions $($DefinitionIds -join ',')."
+    }
+}
+
+Push-Location $PSScriptRoot
+try {
+    $copilotToken = Read-Host "New Copilot GITHUB_TOKEN" -MaskInput
+    if ([string]::IsNullOrWhiteSpace($copilotToken)) {
+        throw "The Copilot GITHUB_TOKEN cannot be empty."
+    }
+
+    Set-AdoAwSecret `
+        -Name "GITHUB_TOKEN" `
+        -Value $copilotToken `
+        -DefinitionIds $copilotDefinitionIds
+
+    Remove-Variable copilotToken
+
+    $issuesToken = Read-Host `
+        "New issues-only GitHub PAT (leave blank to preserve existing issue tokens)" `
+        -MaskInput
+
+    if ([string]::IsNullOrWhiteSpace($issuesToken)) {
+        Write-Host "Preserving existing issue-reporting tokens."
+    }
+    else {
+        Set-AdoAwSecret `
+            -Name "ADO_AW_DEBUG_GITHUB_TOKEN" `
+            -Value $issuesToken `
+            -DefinitionIds $reporterDefinitionIds
+
+        Set-AdoAwSecret `
+            -Name "EXECUTOR_E2E_GITHUB_TOKEN" `
+            -Value $issuesToken `
+            -DefinitionIds $executorDefinitionIds
+
+        Set-AdoAwSecret `
+            -Name "TRIGGER_E2E_GITHUB_TOKEN" `
+            -Value $issuesToken `
+            -DefinitionIds $triggerDefinitionIds
+    }
+
+    Remove-Variable issuesToken
+
+    Write-Host ""
+    Write-Host "Verifying variable names and secret flags (values are never printed)..."
+    & cargo run --quiet -- secrets list `
+        --org $Organization `
+        --project $Project `
+        --definition-ids ($allDefinitionIds -join ",")
+
+    if ($LASTEXITCODE -ne 0) {
+        throw "Secret rotation succeeded, but verification failed."
+    }
+}
+finally {
+    Remove-Variable copilotToken -ErrorAction SilentlyContinue
+    Remove-Variable issuesToken -ErrorAction SilentlyContinue
+    Pop-Location
+}

scripts/ado-script/package-lock.json

@@ -26,6 +26,7 @@
     "build:prepare-pr-base": "ncc build src/prepare-pr-base/index.ts -o .ado-build/prepare-pr-base -m -t && node -e \"const fs=require('node:fs'); fs.copyFileSync('.ado-build/prepare-pr-base/index.js','prepare-pr-base.js'); fs.rmSync('.ado-build/prepare-pr-base',{recursive:true,force:true});\"",
     "build:executor-e2e": "ncc build src/executor-e2e/index.ts -o .ado-build/executor-e2e -m -t && node -e \"const fs=require('node:fs'); fs.mkdirSync('test-bin',{recursive:true}); fs.copyFileSync('.ado-build/executor-e2e/index.js','test-bin/executor-e2e.js'); fs.rmSync('.ado-build/executor-e2e',{recursive:true,force:true});\"",
     "build:trigger-e2e": "ncc build src/trigger-e2e/index.ts -o .ado-build/trigger-e2e -m -t && node -e \"const fs=require('node:fs'); fs.mkdirSync('test-bin',{recursive:true}); fs.copyFileSync('.ado-build/trigger-e2e/index.js','test-bin/trigger-e2e.js'); fs.rmSync('.ado-build/trigger-e2e',{recursive:true,force:true});\"",
+    "build:compiler-smoke-e2e": "ncc build src/compiler-smoke-e2e/index.ts -o .ado-build/compiler-smoke-e2e -m -t && node -e \"const fs=require('node:fs'); fs.mkdirSync('test-bin',{recursive:true}); fs.copyFileSync('.ado-build/compiler-smoke-e2e/index.js','test-bin/compiler-smoke-e2e.js'); fs.rmSync('.ado-build/compiler-smoke-e2e',{recursive:true,force:true});\"",
     "build:check": "ls -lh gate.js && wc -c gate.js",
     "codegen": "node -e \"require('node:fs').mkdirSync('schema', { recursive: true })\" && cargo run --quiet --manifest-path ../../Cargo.toml -- export-gate-schema --output schema/gate-spec.schema.json && npx json2ts schema/gate-spec.schema.json -o src/shared/types.gen.ts --bannerComment \"// AUTO-GENERATED from Rust IR via cargo run -- export-gate-schema. Do not edit; run npm run codegen.\" && cargo run --quiet --manifest-path ../../Cargo.toml -- export-fact-catalog --output src/trigger-e2e/fact-catalog.gen.json",
     "test": "vitest run",
@@ -35,7 +36,8 @@
   },
   "dependencies": {
     "azure-devops-node-api": "^14.1.0",
-    "picomatch": "^4.0.4"
+    "picomatch": "^4.0.4",
+    "yaml": "^2.9.0"
   },
   "devDependencies": {
     "@types/node": "^20.19.39",