test: (scriptless) Enable scriptless phase 3 in AB e2es by lilypan26 · Pull Request #8453 · Azure/AgentBaker

lilypan26 · 2026-05-05T16:25:30Z

What this PR does / why we need it:

Enables scriptless phase 3 in ab e2es under tag EnableScriptlessANC
- This tag means both ANC and NBC cse cmd will be provided, tests should validate that there are no diffs between the generated provisioning env vars
- For now, NBC cse cmd will be used for provisioning until there are no more diffs, after which we can switch to using ANC
Adds CustomDataPhase3 to provide both ANC and NBC cse cmd to AKS node controller

Which issue(s) this PR fixes:

Fixes #

…scriptless/phase-3-e2e

Copilot

Pull request overview

Enables “scriptless phase 3” coverage in the AgentBaker e2e suite by adding a new scriptless_anc subtest path that provisions nodes using AKSNodeConfig/aks-node-controller, plus wiring many existing scenarios to provide an AKSNodeConfigMutator.

Changes:

Added a new scriptless_anc subtest variant and runtime flag (EnableScriptlessANC) to drive scriptless phase-3 execution.
Refactored/expanded the e2e “aks-node-controller hack” customData generation to optionally include AKSNodeConfig and/or an nbc-cmd script.
Updated many scenarios to set equivalent AKSNodeConfigMutator fields alongside existing NBC mutators.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

File	Description
e2e/vmss.go	Refactors customData hack generation and wires scriptless ANC + NBC cmd hack paths into VMSS creation.
e2e/types.go	Adds `EnableScriptlessANC` and adjusts kubelet-config-file detection logic for scriptless ANC scenarios.
e2e/test_helpers.go	Adds `scriptless_anc` subtest generation and new gating helper.
e2e/scenario_test.go	Adds `AKSNodeConfigMutator` coverage across many existing scenarios.

…scriptless/phase-3-e2e

Copilot

Pull request overview

Copilot reviewed 23 out of 29 changed files in this pull request and generated 3 comments.

…scriptless/phase-3-e2e

…e/AgentBaker into lily/scriptless/phase-3-e2e

Copilot

Pull request overview

Copilot reviewed 22 out of 27 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 20 out of 25 changed files in this pull request and generated 4 comments.

Co-authored-by: lilypan26 <106703606+lilypan26@users.noreply.github.com>

…e/AgentBaker into lily/scriptless/phase-3-e2e

Copilot

Pull request overview

Copilot reviewed 21 out of 26 changed files in this pull request and generated 3 comments.

 	var diffs []string
 	for _, key := range sortedKeys {
 		pcVal, inPC := pcEnv[key]
 		nbcVal, inNBC := nbcEnv[key]
 		switch {
 		case inPC && !inNBC:
-			diffs = append(diffs, fmt.Sprintf("only-in-pc: %s", key))
+			diffs = append(diffs, fmt.Sprintf("only-in-pc: %s=%s", key, pcVal))
 		case !inPC && inNBC:
-			diffs = append(diffs, fmt.Sprintf("only-in-nbc: %s", key))
-		case pcVal != nbcVal:
-			diffs = append(diffs, fmt.Sprintf("differs: %s", key))
+			if !isExpectedDiffCSEVar(key) {
+				diffs = append(diffs, fmt.Sprintf("only-in-nbc: %s=%s", key, nbcVal))
+			}
+		case !envValsEqual(pcVal, nbcVal):
+			if !isExpectedDiffCSEVar(key) {
+				diffs = append(diffs, fmt.Sprintf("differs: %s pc=%s nbc=%s", key, pcVal, nbcVal))
+			}


djsly · 2026-06-08T11:20:08Z

AgentBaker Linux PR gate — E2E mass-failure (change-caused, HIGH confidence)

Run: 167125948 (failed)
Failed task: Run AgentBaker E2E → AzureCLI → exit 1
Net new failures: ~13 leaves, all under Test_LocalDNSHostsPlugin*.

Two failure shapes, same root cause:

Loud / fast (1 leaf): Test_LocalDNSHostsPlugin_Scriptless/ACL/default (155.83s) → e2e/validation.go:52 🔴 expected no env var differences between provision-config and nbc-cmd, but found differences: — the new compareEnvs validator this PR added fires with a non-empty diff list.
Slow / derivative (7 leaves): Test_LocalDNSHostsPlugin{,_Scriptless}/{Ubuntu2204,Ubuntu2404,AzureLinuxV3,ACL}/{default,scriptless_nbc} → kube.go:189 after ~742s 🔴 "<pod>" haven't appeared in k8s API server: context deadline exceeded. Identical signature across every distro × variant → systemic node-bring-up degradation, not infra.

Likely root cause ((b) + (c) confirmed, (a) ruled out):

This PR flips nbc.EnableScriptlessNBCCSECmd = true for every non-Scriptless scenario in e2e/node_config.go, and e2e/scenario_test.go strips Tags: { Scriptless: true } from each _Scriptless test and adds a BootstrapConfigMutator — so every LocalDNS scenario now provisions via the new scriptless NBC-CSE-Cmd path, including the default (bash-CSE) variants.
parts/linux/cloud-init/artifacts/aks-node-controller-wrapper.sh was changed from if/elif to two independent ifs, so when both files exist --provision-config and --nbc-cmd are both passed to aks-node-controller provision, which then runs compareEnvs.
e2e/node_config.go::nbcToAKSNodeConfigV1 has no LocalDNSProfile mapping — every LocalDNS occurrence in the PR diff is the unrelated GenerateLocalDNSCoreFile template. The localdns systemd unit therefore never gets configured/started on nodes provisioned via the converter, DNS on the node breaks, debug pods can't reach the API server → the kube.go:189 timeout.
The 2 cosmetic localdns.toml.gtpl diff lines (blank-line removal + trailing newline) are not the cause.

Build-vs-test: test-code-caused (e2e converter + scenario tagging) with a tightly-coupled product-code enabler (baker.go scriptless plumbing + aks-node-controller-wrapper.sh). Not a main regression.

Confidence: HIGH — three independent indicators converge: PR's own validator firing loudly on ACL/default, the EnableScriptlessNBCCSECmd flip mechanism, and the perfectly-symmetric distro×variant timeout pattern.

Strongest alternative (less likely): sysctlTemplateString rewrite in baker.go degrading DNS/conntrack reliability — refuted because every branch still sets tcp_retries2=8, LocalDNS scenarios don't customize sysctls, and a sysctl change wouldn't produce a 100% deterministic per-distro pattern lined up with files this PR touched.

Side-channel (not the cause, FYI): build2204arm64gen2containerd and build2404gen2containerd flagged succeededWithIssues with CIS regressions detected (1) — same 6.1.4.1 pattern recurring across renovate PRs the last 48h (#8652, #8294); non-gating here.

Recommended next action (owner: @lilypan26):

Look at the ACL/default validation.go:52 env-var diff list first — it names the exact env keys the converter mis-maps.
Audit e2e/node_config.go::nbcToAKSNodeConfigV1 for missing NBC field coverage — start with LocalDNSProfile, then any field that drives a systemd unit / cloud-init artifact. Use pkg/agent/baker.go rendering and aks-node-controller/parser as the reference.
Decide if enabling EnableScriptlessNBCCSECmd = true for default variants of every scenario is intended in Phase 3 — if yes, converter coverage must be exhaustive before merge; if not, scope the flip to opted-in scenarios.
Re-run the LocalDNS slice (Ubuntu2204/2404 + AzureLinuxV3 + ACL × {default, scriptless_nbc}) before re-requesting review.

Posted by Clawpilot AgentBaker gate detective.

Copilot

Pull request overview

Copilot reviewed 21 out of 26 changed files in this pull request and generated 2 comments.

lilypan26 added 4 commits April 28, 2026 10:54

enable aksnodeconfig in all e2es

042c0f4

Merge branch 'main' of https://github.com/Azure/AgentBaker into lily/…

ffa89ce

…scriptless/phase-3-e2e

refactor customdatahack and get e2e passing

2c493c8

fix scriptless vs nonscriptless e2e logic

1bb406d

Copilot AI review requested due to automatic review settings May 5, 2026 16:25

lilypan26 requested review from AbelHu, Devinwong, SriHarsha001, awesomenix, calvin197, cameronmeissner, djsly, ganeshkumarashok, junjiezhang1997, mxj220, pdamianov-dev, phealy, r2k1, sulixu, surajssd, timmy-wright and zachary-bailey as code owners May 5, 2026 16:25

lilypan26 temporarily deployed to test May 5, 2026 16:25 — with GitHub Actions Inactive

lilypan26 changed the title ~~Lily/scriptless/phase 3 e2e~~ test(scriptless): Enable scriptless phase 3 in AB e2es May 5, 2026

Copilot started reviewing on behalf of lilypan26 May 5, 2026 16:26 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

Comment thread e2e/vmss.go Outdated

Comment thread e2e/vmss.go Outdated

Comment thread e2e/vmss.go Outdated

Comment thread e2e/vmss.go

Comment thread e2e/test_helpers.go

Comment thread e2e/vmss.go Outdated

restore

02b46dc

lilypan26 temporarily deployed to test May 6, 2026 00:17 — with GitHub Actions Inactive

lilypan26 added 2 commits May 10, 2026 13:38

Merge branch 'main' of https://github.com/Azure/AgentBaker into lily/…

85702a0

…scriptless/phase-3-e2e

add phase 3 custom data and fix e2e logic

dba6573

Copilot started reviewing on behalf of lilypan26 June 5, 2026 00:02 View session

aks-node-assistant Bot added 4 commits June 5, 2026 00:02

chore: tidy aks-node-controller/go.mod

14c4e7c

chore: tidy aks-node-controller/go.sum

f7bb8be

chore: tidy e2e/go.mod

67f6673

chore: tidy e2e/go.sum

4c7177c

aks-node-assistant Bot temporarily deployed to test June 5, 2026 00:03 Inactive

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Comment thread aks-node-controller/pkg/nodeconfigutils/utils.go Outdated

Comment thread e2e/vmss.go Outdated

Comment thread e2e/test_helpers.go

lilypan26 added 2 commits June 4, 2026 22:12

Merge branch 'main' of https://github.com/Azure/AgentBaker into lily/…

364fee8

…scriptless/phase-3-e2e

Merge branch 'lily/scriptless/phase-3-e2e' of https://github.com/Azur…

56078ab

…e/AgentBaker into lily/scriptless/phase-3-e2e

lilypan26 temporarily deployed to test June 5, 2026 05:16 — with GitHub Actions Inactive

aks-node-assistant Bot added 2 commits June 5, 2026 05:18

chore: tidy e2e/go.mod

c5bc723

chore: tidy e2e/go.sum

fd9bb34

Copilot AI review requested due to automatic review settings June 5, 2026 05:18

aks-node-assistant Bot temporarily deployed to test June 5, 2026 05:18 Inactive

lilypan26 added 2 commits June 4, 2026 22:18

go mod tidy

0ca24d8

Merge branch 'lily/scriptless/phase-3-e2e' of https://github.com/Azur…

6850eec

…e/AgentBaker into lily/scriptless/phase-3-e2e

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Comment thread aks-node-controller/pkg/nodeconfigutils/utils_test.go Outdated

Comment thread e2e/test_helpers.go

lilypan26 added 2 commits June 4, 2026 22:26

fix test

20575b6

clean up

510772e

Copilot AI reviewed Jun 5, 2026

View reviewed changes

Comment thread pkg/agent/baker.go Outdated

Comment thread e2e/test_helpers.go

Comment thread aks-node-controller/parser/parser.go

Comment thread aks-node-controller/app.go Outdated

lilypan26 and others added 4 commits June 4, 2026 22:49

linter fix

6ff8a29

Fix supportsScriptlessNBCCSECmd to exclude AKSNodeConfig-based scenarios

463b5e2

Co-authored-by: lilypan26 <106703606+lilypan26@users.noreply.github.com>

fix phase 2.5 e2e logic

84ffc8f

Merge branch 'lily/scriptless/phase-3-e2e' of https://github.com/Azur…

6676c75

…e/AgentBaker into lily/scriptless/phase-3-e2e

Copilot AI reviewed Jun 8, 2026

View reviewed changes

lilypan26 added 2 commits June 8, 2026 11:27

fix test diffs

06abe4e

fix UT

95d2a99

Copilot AI reviewed Jun 8, 2026

View reviewed changes

Comment thread parts/linux/cloud-init/artifacts/aks-node-controller-wrapper.sh

Comment thread parts/linux/cloud-init/artifacts/aks-node-controller-wrapper.sh

Conversation

lilypan26 commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

djsly commented Jun 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

lilypan26 commented May 5, 2026 •

edited

Loading