debug: candidate smoke misreports invalid policy response as drift

[jamesadevine]

Diagnostic report

Target run

• Pipeline: ado-aw candidate compiler smoke (definition 2559)
• Build: 626732
• Build number: 20260724.11
• Trigger: GitHub PR comment /azp run ado-aw candidate compiler smoke
• Reason/source: pullRequest, refs/pull/1508/merge
• PR head: addf62e7d3986f23364d9b98b9fb792f0cd3daa9
• Built merge commit: 779ec5f1d0495537a045d9bc20a5bf6377f3a7b6

Summary

The manual comment trigger worked and attached the optional GitHub check to PR #1508. The orchestrator built and published the candidate compiler artifact, but aborted before queueing any of the five child smoke pipelines.

First failure

The first failed timeline record was Audit AgentPlayground trigger policy (log 18):

parse error: Invalid numeric literal at line 1, column 26
Candidate compiler PR comment-gate policy drift detected for definition 2559.
Bash exited with code '1'.

The new self-audit performs a second REST GET for definition 2559, stores the body in SELF_JSON, and pipes it to jq. The response received during this request was not valid JSON. The if ! ... jq branch treats every parse/runtime error as a failed policy predicate, so it reported policy drift even though the predicate was never evaluated successfully.

Policy state

The live definition is revision 6 and currently contains all required top-level settings:

• isCommentRequiredForPullRequest: true
• requireCommentsForNonTeamMembersOnly: false
• requireCommentsForNonTeamMemberAndNonContributors: false
• isCommentRequiredForInternalRepoPRs: true
• commentOptionInternalRepos: all

A fresh REST query returns valid JSON with those values. Therefore the evidence does not show actual comment-gate drift. The exact malformed response cannot be recovered because the step did not retain its body, status metadata, or content type.

Secondary failure

Run all candidate compiler smoke pipelines was skipped after the audit failed. That step normally creates $(Build.ArtifactStagingDirectory)/compiler-smoke-diagnostics. The unconditional Publish candidate smoke diagnostics task then failed separately because that directory did not exist:

Path does not exist: /mnt/vss/_work/1/a/compiler-smoke-diagnostics

This added a second error while providing no diagnostic artifact for the primary failure.

Impact

• The manual trigger mechanism itself is validated.
• No candidate child pipelines were queued.
• No compiler, Agent, Detection, SafeOutputs, Azure CLI, or custom-safe-output behavior was exercised.
• The failed check cannot be interpreted as a candidate compiler regression.

Classification

Candidate-smoke harness reliability bug: a malformed/transient definition response is misclassified as policy drift, and the failure path does not preserve diagnostics.

Relevant source

tests/compiler-smoke-e2e/azure-pipelines.yml, in Audit AgentPlayground trigger policy and the following diagnostics publication steps, introduced/extended by addf62e7.

[jamesadevine]

Confirmed root cause after inspecting the committed source: the self-audit added in addf62e7 literally contained:

-H "Authorization: ******"

The surrounding pre-existing definition requests correctly used the bearer token. This explains why the earlier PR-definition checks succeeded, while the new redundant self-definition request received a non-JSON authentication response and failed in jq.

The follow-up change removes the duplicate request, reuses the already-authenticated response for definition 2559, rejects literal masked headers in the policy test, and adds bounded diagnostics containing curl exit/status/content metadata, whitelisted response headers, jq errors, and a response sample. Raw bodies and raw headers stay under $(Agent.TempDirectory) and are never published. The diagnostics artifact directory is initialized before the build/audit steps so condition: always() can publish it on failures.

[jamesadevine]

Resolved by 876a5e74 and validated by manually triggered candidate build 626947 on PR head 58621ee8. The optional check started 17 seconds after the /azp run comment and completed successfully in 28m05s.