ci(prcheck): use 'azldev component changed' for affected-component detection

Replace the bash 'git diff | grep /sources' step with structured azldev
commands for lock validation, change detection, and scoped rendering.

Pipeline step order:
  1. Lock check -- 'azldev component update -a -q -O json', fail if any
     component has changed == true. Lock-update JSON published as a
     pipeline artifact for triage.
  2. Changed-component detection -- 'azldev component changed
     --include-unchanged' writes the full per-component JSON to disk,
     published via ob_outputDirectory. The --include-unchanged flag
     ensures the JSON contains every known component, which is needed
     for the renderable-set filter in step 4.
  3. Source/identity consistency tripwire -- hard-fail if any component
     reports sourcesChange == true with a changeType not in the
     allow-list {added, changed, deleted}. Prevents unauthenticated
     rewrites of the rendered 'sources' file under an existing
     component's identity. Data path is severed (subsequent steps
     skip); PR check remains advisory until ADO task 19179 removes
     job-level continueOnError.
  4. Scoped render -- render set is the union of components flagged by
     'azldev component changed' (inputs differ) and components whose
     spec tree was touched directly in the PR (git diff under specs/,
     mapped back to component names by compute_render_set.py). Deleted
     and unknown components are excluded via a renderable-set filter
     built from the full --include-unchanged JSON.
  5. Prcheck API -- switches from --components <csv> to
     --changed-components-file <path>, filtering to entries with
     sourcesChange == true and changeType in {added, changed}
     (allow-list, mirroring the consistency tripwire).

Also:
* Add --changed-components-file flag (mutually exclusive with
  --components) and _load_components_from_file() to run_prcheck.py.
  Uses an allow-list of changeType values for defense-in-depth.
* Add compute_render_set.py for render-set computation.
* Document AZLDEV_ALLOW_ROOT in ADO pipeline instructions (OneBranch
  containers run as root, azldev refuses by default).
* Mark changedComponentsFile pipeline variable as isreadonly=true.
* Switch API_BASE_URL to $(ApiBaseDirectUrl) (bypasses AFD).

Author: dmcilvaney

.github/instructions/ado-pipeline.instructions.md

@@ -175,6 +175,17 @@ Avoid shell scripts beyond the smallest possible wiring (env exports, `##vso[...
 
 Python scripts are easier to test locally, easier to review, and avoid the foot-guns of bash quoting / globbing.
 
+### `azldev` in OneBranch
+
+OneBranch run all steps as `root`. `azldev` refuses to run many commands as root by default (a safety measure for developer workstations). To allow it in CI, set the environment variable `AZLDEV_ALLOW_ROOT=1` if azldev returns the error `ERR Error: this command may not be run as root`. This is NOT safe for general use, only for use in disposable CI environments. Set it in the `env:` block of the step, not inline in the script body.
+
+```yaml
+env:
+  # OneBranch containers run as root. azldev refuses to run as root
+  # by default, disable the root security check for this step.
+  AZLDEV_ALLOW_ROOT: "1"
+```
+
 ## Security hardening
 
 Apply all of these unless there is a documented reason not to:

.github/workflows/ado/templates/sources-upload-stages.yml

@@ -120,23 +120,163 @@ stages:
           - script: |
               set -euo pipefail
 
-              # Workaround for an ADO git config error during spec rendering.
+              # Workaround for an ADO git config error.
               # The config key may not be present on every agent image, so tolerate its absence.
               git config --unset extensions.worktreeConfig || true
 
-              # Full history is needed for spec rendering to work.
+              # Full history is needed for lock resolution and spec rendering.
               if [ "$(git rev-parse --is-shallow-repository)" = "true" ]; then
                 echo "##[group]Fetching full git history"
                 git fetch --unshallow
                 echo "##[endgroup]"
               fi
 
-              specs_dir="$(azldev config dump -q -f json | jq -r '.project.renderedSpecsDir')"
+              echo "##[group]Updating lock files"
+              lock_update_file="$ARTIFACT_STAGING_DIR/lock-update.json"
+              azldev component update -a -q -O json > "$lock_update_file"
+              echo "##[endgroup]"
+
+              # Also copy into ob_outputDirectory so OneBranch auto-publishes the
+              # lock-update artifact for post-mortem triage.
+              ob_lock_dir="$OB_OUTPUT_DIR/lock-update"
+              mkdir -p "$ob_lock_dir"
+              cp "$lock_update_file" "$ob_lock_dir/"
+
+              # Check if any locks drifted -- azldev tells us directly.
+              # If locks are not valid, we can't guarantee that any further steps are valid, so fail the PR check immediately.
+              # azldev emits a JSON 'null' when nothing changed; '. // []' coerces that to an
+              # empty array so the downstream '[.[] | ...]' pipeline is well-typed.
+              drifted=$(jq '(. // []) | [.[] | select(.changed == true)] | length' "$lock_update_file")
+              if [[ "$drifted" -gt 0 ]]; then
+                echo "##[error]$drifted lock file(s) are not up to date. Run 'azldev component update -a' and commit the result."
+                echo "##[group]Drifted components"
+                jq -r '(. // []) | .[] | select(.changed == true) | .component' "$lock_update_file"
+                echo "##[endgroup]"
+                exit 1
+              fi
+            displayName: "Verify lock files are up to date"
+            env:
+              ARTIFACT_STAGING_DIR: $(Build.ArtifactStagingDirectory)
+              # OneBranch containers run as root. azldev refuses to run as root
+              # by default, disable the root security check for this step.
+              AZLDEV_ALLOW_ROOT: "1"
+              OB_OUTPUT_DIR: $(ob_outputDirectory)
+
+          - script: |
+              set -euo pipefail
+
+              artifact_dir="$ARTIFACT_STAGING_DIR/changed-components"
+              json_file="$artifact_dir/changed-components.json"
+              mkdir -p "$artifact_dir"
+
+              # 'azldev component changed' compares the source and target commits and emits one
+              # entry per component with its 'changeType' ('added', 'changed', 'unchanged', or
+              # 'deleted') plus a 'sourcesChange' flag indicating whether the rendered 'sources'
+              # file differs between commits. Only components with sourcesChange == true AND
+              # changeType != 'deleted' are forwarded to Control Tower for upload.
 
-              echo "##[group]Specs rendering"
-              azldev component render -q -a --clean-stale
+              echo "##[group]Computing changed components"
+              azldev component changed --from "$TARGET_COMMIT" --to "$SOURCE_COMMIT" -a --include-unchanged -O json > "$json_file"
               echo "##[endgroup]"
 
+              echo "##[group]Changed components (non-unchanged only)"
+              jq '(. // []) | [.[] | select(.changeType != "unchanged")]' "$json_file"
+              echo "##[endgroup]"
+
+              echo "##[group]Upload set (sourcesChange == true, changeType in {added, changed})"
+              upload_count=$(jq -r '(. // []) | [.[] | select(.sourcesChange == true and (.changeType | IN("added", "changed")))] | length' "$json_file")
+              jq -r '(. // []) | .[] | select(.sourcesChange == true and (.changeType | IN("added", "changed"))) | .component' "$json_file" | sort
+              echo "Total: $upload_count component(s) to upload."
+              echo "##[endgroup]"
+
+              # Also write into ob_outputDirectory so OneBranch auto-publishes it
+              # as a build artifact (explicit PublishPipelineArtifact is forbidden).
+              ob_dir="$OB_OUTPUT_DIR/changed-components"
+              mkdir -p "$ob_dir"
+              cp "$json_file" "$ob_dir/"
+
+              echo "##vso[task.setvariable variable=changedComponentsFile;isreadonly=true]$json_file"
+            env:
+              ARTIFACT_STAGING_DIR: $(Build.ArtifactStagingDirectory)
+              # OneBranch containers run as root. azldev refuses to run as root
+              # by default, disable the root security check for this step.
+              AZLDEV_ALLOW_ROOT: "1"
+              OB_OUTPUT_DIR: $(ob_outputDirectory)
+              SOURCE_COMMIT: $(sourceCommit)
+              TARGET_COMMIT: $(targetCommit)
+            displayName: "Compute changed components"
+
+          - script: |
+              set -euo pipefail
+
+              # Source/identity consistency tripwire. The rendered 'sources' file
+              # describes the lookaside tarballs Control Tower will fetch and serve to
+              # builds. If a PR mutates that file but the component's input fingerprint
+              # is unchanged, we cannot tell from this side whether that's a hostile
+              # supply-chain swap or an accidental hand-edit / renderer non-determinism
+              # bug. We treat any such record as hostile-by-default and hard-fail, so
+              # the new bytes never reach the lookaside under an existing identity.
+              #
+              # The legitimate cases are explicitly enumerated as an allow-list (added,
+              # changed, deleted) so any future 'changeType' value -- e.g. 'renamed'
+              # that doesn't pull the sources blob into the input fingerprint -- fails
+              # closed until the policy here is reviewed.
+              bogus_count=$(jq -r '(. // []) | [.[] | select(.sourcesChange == true and (.changeType | IN("added", "changed", "deleted") | not))] | length' "$CHANGED_COMPONENTS_FILE")
+              if [[ "$bogus_count" -gt 0 ]]; then
+                echo "##[error]$bogus_count component(s) report a sources change without an accompanying identity change. Refusing to forward to Control Tower."
+                jq -r '(. // []) | .[] | select(.sourcesChange == true and (.changeType | IN("added", "changed", "deleted") | not)) | "  - " + .component + " (changeType=" + (.changeType // "null") + ")"' "$CHANGED_COMPONENTS_FILE"
+                exit 1
+              fi
+              echo "OK: every sources change is accompanied by an identity change."
+            # When this step fails, subsequent steps in the job are skipped -- the
+            # bad data never reaches Control Tower. However, the job-level
+            # continueOnError: true means the PR check still reports green to
+            # GitHub. The consistency tripwire becomes truly blocking once ADO
+            # task 19179 removes the job-level flag.
+            env:
+              CHANGED_COMPONENTS_FILE: $(changedComponentsFile)
+            displayName: "Source/identity consistency check"
+
+          - script: |
+              set -euo pipefail
+
+              # azldev's renderedSpecsDir is absolute. Translate to repo-relative
+              # so it matches git's output ('git diff --name-only' always emits
+              # repo-relative paths regardless of the path arg form).
+              specs_dir_abs="$(azldev config dump -q -f json | jq -r '.project.renderedSpecsDir')"
+              specs_dir="$(realpath --relative-to="$(pwd)" "$specs_dir_abs")"
+
+              # Capture git diff under the specs tree so the render set can
+              # include components whose specs were edited directly (which
+              # azldev's input-fingerprint view of "changed" would miss).
+              # --no-renames prevents collapse of delete+add into a rename
+              # entry which would lose the old path. The Python script
+              # filters out deleted/unknown components using the full
+              # changed-components JSON.
+              specs_diff_file="$ARTIFACT_STAGING_DIR/specs-diff.txt"
+              git diff --no-renames --name-only "$TARGET_COMMIT" "$SOURCE_COMMIT" -- "$specs_dir" > "$specs_diff_file"
+
+              # Render set is the union of:
+              #   - components flagged by 'azldev component changed' (inputs differ)
+              #   - components whose spec tree was touched directly in the PR
+              changed=$(python3 .github/workflows/scripts/control-tower/compute_render_set.py \
+                --changed-components-file "$CHANGED_COMPONENTS_FILE" \
+                --specs-diff-file "$specs_diff_file" \
+                --specs-dir "$specs_dir")
+
+              if [[ -z "$changed" ]]; then
+                echo "No changed components -- skipping render."
+              else
+                changed_count=$(echo "$changed" | wc -l)
+                echo "Rendering $changed_count component(s) (azldev dedupes internally)..."
+                echo "##[group]Render set"
+                echo "$changed" | sed 's/^/  - /'
+                echo "##[endgroup]"
+                echo "##[group]Specs rendering"
+                printf '%s\n' "$changed" | xargs -d '\n' azldev component render -q --
+                echo "##[endgroup]"
+              fi
+
               # Check for any new or modified files under the specs directory.
               if ! python3 .github/workflows/scripts/render-specs-check/check_rendered_specs.py --specs-dir "$specs_dir"; then
                 echo "##[group]Git diff"
@@ -145,51 +285,15 @@ stages:
                 echo "##[error]Specs are not up to date. Run 'azldev component render -q -a --clean-stale' and try again."
                 exit 1
               fi
-            displayName: "Verify rendered specs are up to date"
-            # TODO: Re-enabled failing once specs in the main branch are updated and stable.
-            # ADO task: 19179
-            continueOnError: true
-
-          - script: |
-              set -euo pipefail
-
-              cat << EOF
-              NOTE: It's possible more components changed between the source and target commits than displayed below.
-                    This workflow only checks for updates in the 'sources' files and ignores all other changes.
-                    Only components present on the source commit are reported; components that exist only on the
-                    target branch are intentionally excluded. Pure renames are reported under their source-branch
-                    name, since the component name is part of the cache blob name we upload.
-              EOF
-
-              # Diff direction is TARGET -> SOURCE so additions/modifications represent the source-side state:
-              #   * --no-renames decomposes a rename into an add (new path, source side) + delete (old path,
-              #     target side); we want the source-side name because the component name is part of the cache
-              #     blob name uploaded for the PR. It also silences git's diff.renameLimit warning on large diffs.
-              #   * --diff-filter=d (lowercase) excludes deletions, which here correspond to paths present only
-              #     on the target branch -- those components are not on the PR and must not be uploaded.
-              # Net effect: the listed '*/sources' files are exactly those present on the source commit that
-              # differ from target, including pure renames surfaced under the source-branch name.
-              sources_files="$(
-                git diff --no-renames --diff-filter=d "$TARGET_COMMIT" "$SOURCE_COMMIT" --name-only \
-                  | { grep '/sources$' || true; } \
-                  | sort -u
-              )"
-
-              components=()
-              while IFS= read -r line; do
-                # Skip the spurious empty iteration when no '*/sources' files changed.
-                if [[ -n "$line" ]]; then
-                  components+=("$(basename "$(dirname "$line")")")
-                fi
-              done <<< "$sources_files"
-
-              components="$(IFS=, ; echo "${components[*]}")"
-              echo "Affected components: $components"
-              echo "##vso[task.setvariable variable=components]$components"
             env:
+              ARTIFACT_STAGING_DIR: $(Build.ArtifactStagingDirectory)
+              # OneBranch containers run as root. azldev refuses to run as root
+              # by default, disable the root security check for this step.
+              AZLDEV_ALLOW_ROOT: "1"
+              CHANGED_COMPONENTS_FILE: $(changedComponentsFile)
               SOURCE_COMMIT: $(sourceCommit)
               TARGET_COMMIT: $(targetCommit)
-            displayName: "Determine affected components"
+            displayName: "Render changed specs and verify rendered tree"
 
           - task: AzureCLI@2
             displayName: "Call Control Tower 'prcheck' API"
@@ -204,14 +308,14 @@ stages:
                   --api-audience "$API_AUDIENCE" \
                   --api-base-url "$API_BASE_URL" \
                   --build-reason "$BUILD_REASON" \
-                  --components "$COMPONENTS" \
+                  --changed-components-file "$CHANGED_COMPONENTS_FILE" \
                   --source-commit "$SOURCE_COMMIT" \
                   --repo-uri "$UPSTREAM_REPO_URL"
             env:
               API_AUDIENCE: $(ApiAudience)
               API_BASE_URL: $(ApiBaseDirectUrl)
               BUILD_REASON: $(Build.Reason)
-              COMPONENTS: $(components)
+              CHANGED_COMPONENTS_FILE: $(changedComponentsFile)
               SOURCE_COMMIT: $(sourceCommit)
               # TODO: Target commit is not used. Will be needed once we move detection of affected components to CT.
               # ADO task: 18816

.github/workflows/scripts/control-tower/compute_render_set.py

@@ -0,0 +1,84 @@
+"""Compute the render set: components flagged by `azldev component changed`
+plus components whose spec tree was touched directly in the PR.
+
+Emits one component name per line on stdout (azldev dedupes internally).
+"""
+
+import argparse
+import json
+from pathlib import Path
+
+
+def _load_entries(path: Path) -> list[dict]:
+    """Load the changed-components JSON, coercing null to []."""
+    return json.loads(path.read_text(encoding="utf-8")) or []
+
+
+def _renderable_components(entries: list[dict]) -> set[str]:
+    """Component names that azldev can still render (everything except deleted)."""
+    return {
+        e["component"]
+        for e in entries
+        if e.get("changeType") != "deleted"
+    }
+
+
+def from_changed(entries: list[dict]) -> list[str]:
+    """Components from `azldev component changed` JSON.
+
+    Includes anything that is not 'deleted' and either has a non-'unchanged'
+    changeType or has sourcesChange=true (safety net).
+    """
+    out = []
+    for e in entries:
+        change_type = e.get("changeType")
+        sources_change = e.get("sourcesChange") is True
+        if change_type == "deleted":
+            continue
+        if change_type == "unchanged" and not sources_change:
+            continue
+        out.append(e["component"])
+    return out
+
+
+def from_specs_diff(path: Path, specs_dir: Path, renderable: set[str]) -> list[str]:
+    """Components with any modified file under specs_dir.
+
+    Spec layout is rigid: <specs_dir>/<first-char>/<component>/...
+    so the component name is the second segment under specs_dir.
+
+    Only components in ``renderable`` are included -- a deleted component's
+    .comp.toml is gone so ``azldev component render`` would fail, and a
+    path that doesn't map to any known component is noise.
+    """
+    prefix = str(specs_dir).rstrip("/") + "/"
+    out = []
+    for line in path.read_text(encoding="utf-8").splitlines():
+        if not line.startswith(prefix):
+            continue
+        parts = line[len(prefix):].split("/", 2)
+        if len(parts) >= 2 and parts[1]:
+            name = parts[1]
+            if name in renderable:
+                out.append(name)
+    return out
+
+
+def main() -> None:
+    p = argparse.ArgumentParser()
+    p.add_argument("--changed-components-file", type=Path, required=True)
+    p.add_argument("--specs-diff-file", type=Path, required=True)
+    p.add_argument("--specs-dir", type=Path, required=True)
+    args = p.parse_args()
+
+    entries = _load_entries(args.changed_components_file)
+    renderable = _renderable_components(entries)
+
+    for name in from_changed(entries):
+        print(name)
+    for name in from_specs_diff(args.specs_diff_file, args.specs_dir, renderable):
+        print(name)
+
+
+if __name__ == "__main__":
+    main()