Skip to content

Commit cb03e39

Browse files
jongioCopilotrajeshkamal5050
authored
TME test config, mage record command, and re-recorded PreflightQuota cassettes (#7779)
* Support azd config defaults.test.* keys for functional test config Update cliConfig.init() resolution order to check test-specific azd config keys (defaults.test.subscription, defaults.test.tenant, defaults.test.location) before falling back to general defaults (defaults.subscription, defaults.location). This enables a one-time setup for developers: azd config set defaults.test.subscription <TME-SUB-ID> azd config set defaults.test.tenant <TME-TENANT-ID> azd config set defaults.test.location eastus2 No more manual env var exports per session. Fixes #7777 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: add mage record and defaults.test.* config documentation Update CONTRIBUTING.md, AGENTS.md, recording guide, and environment variables docs to document the new mage record command and defaults.test.* azd config keys for test subscription configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * exclude stale recordings from preflight playback VsServer, Deploy_SlotDeployment, and Up_Down_ContainerAppJob recordings are stale and require TME subscription access to re-record. Exclude them from automatic playback so preflight passes for contributors without TME access. Tracked in #7780 and #7014. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * revert: restore original PreflightQuota recordings The cassettes I re-recorded in 519138c used my personal subscription ID (25fd0362-...) instead of the TME subscription (4d042dc6-...) that CI expects. This caused 'requested interaction not found' errors on all 3 build platforms because azd in CI generates URLs with the TME sub but the cassette URLs use my personal sub. Reverting these 6 cassettes back to the originals from fe57152 (which were recorded against TME and worked in CI). Note: re-recording these requires TME subscription access (see #7780). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * address review feedback: simplify mage record, clarify maintainer workflow - magefile.go: remove manual go build; delegate to azdcli.NewCLI's buildRecordOnce (addresses weikanglim feedback) - AGENTS.md / CONTRIBUTING.md / recording-functional-tests-guide.md: clarify re-recording is a core-maintainer workflow; any subscription works Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address Copilot PR review feedback - magefile.go: only apply excludedPlaybackTests in playback mode so `mage record -filter=<name>` can re-record stale tests (the whole point of excluding them was to enable re-recording) - magefile.go: include opts.mode in 'no tests match filter' message - cli_test.go: clarify config fallbacks only apply when CI is unset - environment-variables.md: note AZD_TEST_TENANT_ID has no defaults.tenant global fallback; clarify CI-gating - recording-functional-tests-guide.md: same doc clarifications Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Re-record PreflightQuota cassettes against TME subscription All 6 preflight quota tests re-recorded and verified in both record and playback modes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address wbreza review feedback - Extract configFallback helper in cli_test.go to remove DRY violation across subscription/tenant/location config lookups - Add opts.mode validation in runFunctionalTests to make the record/playback contract explicit Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Rajesh Kamal <rajeshkamal@microsoft.com>
1 parent 36160ee commit cb03e39

12 files changed

Lines changed: 1855 additions & 736 deletions

cli/azd/AGENTS.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,10 @@ When writing tests, prefer table-driven tests. Use testify/mock for mocking.
7575

7676
> **Tip**: The `/azd-preflight` Copilot skill runs all these checks and auto-fixes issues. See `.github/skills/azd-preflight/`.
7777
78+
Additional mage targets:
79+
80+
- `mage record` — re-record functional test cassettes against a live Azure subscription. Accepts an optional `-filter=TestName` flag to re-record specific tests. Typically only core maintainers need to run this; external contributors can rely on playback mode (the default) which requires no Azure access. Requires `azd auth login` and a configured test subscription (see `docs/recording-functional-tests-guide.md`).
81+
7882
```bash
7983
gofmt -s -w .
8084
go fix ./...

cli/azd/CONTRIBUTING.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,27 @@ mage preflight
5959

6060
> **Tip**: If you're using GitHub Copilot, the `/azd-preflight` skill runs `mage preflight` and auto-fixes any issues it discovers.
6161
62+
### Re-recording functional test cassettes
63+
64+
> **Note**: Re-recording is typically a core-maintainer workflow and requires an Azure subscription. External contributors can run the tests in the default playback mode, which replays stored cassettes and needs no Azure access.
65+
66+
Re-record stale functional test recordings against a live Azure subscription:
67+
68+
```bash
69+
cd cli/azd
70+
mage record # re-record all playback tests
71+
mage record -filter=Test_CLI_Quota # re-record only matching tests
72+
```
73+
74+
Core maintainers configure the test subscription and tenant once via `azd config`:
75+
76+
```bash
77+
azd config set defaults.test.subscription <SUBSCRIPTION_ID>
78+
azd config set defaults.test.tenant <TENANT_ID>
79+
```
80+
81+
These values are stored in your user-level azd config and persist across sessions. You can also set them via environment variables (`AZD_TEST_AZURE_SUBSCRIPTION_ID`, `AZD_TEST_TENANT_ID`), which take precedence. Any Azure subscription you have access to works; the `defaults.test.*` namespace is kept separate from the top-level `defaults.*` so test config does not affect regular `azd` usage. See the [recording guide](./docs/recording-functional-tests-guide.md) for details.
82+
6283
Run tests:
6384

6485
```bash

cli/azd/docs/environment-variables.md

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -230,13 +230,28 @@ These variables are used by the Terraform provider integration to authenticate w
230230
## Test Variables
231231

232232
> **Warning**: Test variables are used by the `azd` test suite only and are not intended for end users.
233-
234-
| Variable | Description |
235-
| --- | --- |
236-
| `AZD_TEST_CLIENT_ID` | The client ID for test authentication. |
237-
| `AZD_TEST_TENANT_ID` | The tenant ID for test authentication. |
238-
| `AZD_TEST_AZURE_SUBSCRIPTION_ID` | The Azure subscription ID for tests. |
239-
| `AZD_TEST_AZURE_LOCATION` | The Azure location for tests. |
233+
>
234+
> **Tip**: Instead of setting environment variables for every session, you can persist test defaults
235+
> in your user-level `azd` config. These config keys act as fallbacks when the corresponding
236+
> environment variable is not set:
237+
>
238+
> ```bash
239+
> azd config set defaults.test.subscription <SUBSCRIPTION_ID>
240+
> azd config set defaults.test.tenant <TENANT_ID>
241+
> azd config set defaults.test.location <LOCATION>
242+
> ```
243+
>
244+
> Resolution order: environment variable → `defaults.test.*``defaults.*` (global default).
245+
> Note: `AZD_TEST_TENANT_ID` only falls back to `defaults.test.tenant` (no
246+
> `defaults.tenant` global fallback). Config fallbacks are only consulted when
247+
> the `CI` environment variable is unset.
248+
249+
| Variable | Description | Config Fallback |
250+
| --- | --- | --- |
251+
| `AZD_TEST_CLIENT_ID` | The client ID for test authentication. ||
252+
| `AZD_TEST_TENANT_ID` | The tenant ID for test authentication. | `defaults.test.tenant` |
253+
| `AZD_TEST_AZURE_SUBSCRIPTION_ID` | The Azure subscription ID for tests. | `defaults.test.subscription` |
254+
| `AZD_TEST_AZURE_LOCATION` | The Azure location for tests. | `defaults.test.location` |
240255
| `AZD_TEST_CLI_VERSION` | Overrides the CLI version reported during tests. |
241256
| `AZD_TEST_FIXED_CLOCK_UNIX_TIME` | Sets a fixed clock time (Unix epoch) for deterministic tests. |
242257
| `AZD_TEST_HTTPS_PROXY` | The HTTPS proxy URL for tests. |

cli/azd/docs/recording-functional-tests-guide.md

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -262,7 +262,42 @@ This helper (from `test/functional/aspire_test.go`) deletes subscription deploym
262262

263263
## Re-recording an Existing Test
264264

265-
### Local Development
265+
> **Note**: Re-recording is typically a core-maintainer workflow. External contributors generally do not need to re-record cassettes; the default playback mode replays stored interactions and requires no Azure access. Open an issue if a cassette appears stale.
266+
267+
### Using `mage record` (Recommended)
268+
269+
The easiest way to re-record tests is the `mage record` target:
270+
271+
```bash
272+
cd cli/azd
273+
mage record -filter=Test_CLI_MyNewFeature # re-record a specific test
274+
mage record # re-record all playback tests
275+
```
276+
277+
`mage record` handles building the `azd-record` binary, setting `AZURE_RECORD_MODE=record`, and running the test with a 30-minute timeout.
278+
279+
#### Configuring the Test Subscription
280+
281+
Tests need an Azure subscription and tenant. Any subscription you have access to works — the `defaults.test.*` keys are a separate namespace so test configuration does not affect regular `azd` defaults. Configure them once with `azd config` (persists across sessions):
282+
283+
```bash
284+
azd config set defaults.test.subscription <SUBSCRIPTION_ID>
285+
azd config set defaults.test.tenant <TENANT_ID>
286+
```
287+
288+
The resolution order is: environment variable → `defaults.test.*` config → `defaults.*` config.
289+
Note: tenant only falls back to `defaults.test.tenant` (no `defaults.tenant` global fallback).
290+
Config fallbacks are only consulted when the `CI` environment variable is unset.
291+
292+
| Setting | Environment Variable | Config Key |
293+
|---------|---------------------|------------|
294+
| Subscription | `AZD_TEST_AZURE_SUBSCRIPTION_ID` | `defaults.test.subscription` |
295+
| Tenant | `AZD_TEST_TENANT_ID` | `defaults.test.tenant` |
296+
| Location | `AZD_TEST_AZURE_LOCATION` | `defaults.test.location` |
297+
298+
### Manual Re-recording
299+
300+
If you prefer manual control:
266301

267302
1. **Delete existing recording**:
268303
```bash

cli/azd/magefile.go

Lines changed: 140 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -123,28 +123,13 @@ func (Dev) Uninstall() error {
123123
//
124124
// Usage: mage preflight
125125
func Preflight() error {
126-
// Disable Go workspace mode so preflight mirrors CI, which has no go.work file.
127-
// Without this, a local go.work can silently resolve different module versions
128-
// than go.mod alone, masking build failures that only appear in CI.
129-
defer setEnvScoped("GOWORK", "off")()
130-
131-
repoRoot, err := findRepoRoot()
126+
azdDir, cleanup, err := mageInit()
132127
if err != nil {
133128
return err
134129
}
135-
azdDir := filepath.Join(repoRoot, "cli", "azd")
130+
defer cleanup()
136131

137-
// Pin GOTOOLCHAIN to the version declared in go.mod when it isn't already
138-
// set. When the system Go is older (e.g. 1.25) and go.mod says 1.26,
139-
// parallel compilations can race the auto-download, producing "compile:
140-
// version X does not match go tool version Y" errors. Pinning upfront
141-
// avoids this. We skip the override when GOTOOLCHAIN is already set so
142-
// that a user's explicit choice (or a newer Go) is respected.
143-
if _, hasToolchain := os.LookupEnv("GOTOOLCHAIN"); !hasToolchain {
144-
if ver, err := goModVersion(azdDir); err == nil && ver != "" {
145-
defer setEnvScoped("GOTOOLCHAIN", "go"+ver)()
146-
}
147-
}
132+
repoRoot := filepath.Dir(filepath.Dir(azdDir))
148133

149134
// Check required tools are installed before running anything.
150135
if err := requireTool("golangci-lint",
@@ -381,8 +366,10 @@ func Preflight() error {
381366

382367
// 8. Playback tests
383368
wg2.Go(func() {
384-
playbackEnv := append(skipBuildEnv, "AZURE_RECORD_MODE=playback")
385-
if err := runPlaybackTestsWithEnv(azdDir, playbackEnv); err != nil {
369+
if err := runFunctionalTests(azdDir, testRunOpts{
370+
mode: "playback",
371+
env: skipBuildEnv,
372+
}); err != nil {
386373
results[checkPlayback] = checkResult{"fail", err.Error(), ""}
387374
} else {
388375
results[checkPlayback] = checkResult{"pass", "", ""}
@@ -424,69 +411,167 @@ func Preflight() error {
424411
//
425412
// Usage: mage playbackTests
426413
func PlaybackTests() error {
427-
defer setEnvScoped("GOWORK", "off")()
414+
azdDir, cleanup, err := mageInit()
415+
if err != nil {
416+
return err
417+
}
418+
defer cleanup()
428419

429-
repoRoot, err := findRepoRoot()
420+
return runFunctionalTests(azdDir, testRunOpts{mode: "playback"})
421+
}
422+
423+
// Record re-records functional test playback cassettes against live Azure.
424+
// Usage:
425+
//
426+
// mage record # re-record ALL playback tests
427+
// mage record -filter=TestName # re-record tests matching filter
428+
//
429+
// Requires:
430+
// - Azure authentication (azd auth login)
431+
// - Test subscription config (azd config set defaults.test.subscription/tenant/location)
432+
// OR equivalent AZD_TEST_* environment variables
433+
//
434+
// The azd-record binary (built with -tags=record) is built on demand by
435+
// azdcli.NewCLI via buildRecordOnce when tests run in record mode. To use a
436+
// custom pre-built binary, set CLI_TEST_AZD_PATH before invoking mage record.
437+
func Record(filter *string) error {
438+
azdDir, cleanup, err := mageInit()
430439
if err != nil {
431440
return err
432441
}
433-
azdDir := filepath.Join(repoRoot, "cli", "azd")
442+
defer cleanup()
443+
444+
return runFunctionalTests(azdDir, testRunOpts{
445+
mode: "record",
446+
filter: filter,
447+
verbose: true,
448+
})
449+
}
450+
451+
// mageInit sets up the standard mage environment (GOWORK=off, GOTOOLCHAIN pin)
452+
// and returns the cli/azd directory path. Callers must defer the returned
453+
// cleanup function to restore environment variables.
454+
func mageInit() (azdDir string, cleanup func(), err error) {
455+
restoreGowork := setEnvScoped("GOWORK", "off")
434456

435-
// Pin GOTOOLCHAIN (see Preflight for rationale).
457+
repoRoot, err := findRepoRoot()
458+
if err != nil {
459+
restoreGowork()
460+
return "", nil, err
461+
}
462+
azdDir = filepath.Join(repoRoot, "cli", "azd")
463+
464+
// Pin GOTOOLCHAIN to the version declared in go.mod when it isn't already
465+
// set. Prevents parallel compilation races when the system Go version
466+
// differs from go.mod (see Preflight for full rationale).
467+
var restoreToolchain func()
436468
if _, hasToolchain := os.LookupEnv("GOTOOLCHAIN"); !hasToolchain {
437469
if ver, err := goModVersion(azdDir); err == nil && ver != "" {
438-
defer setEnvScoped("GOTOOLCHAIN", "go"+ver)()
470+
restoreToolchain = setEnvScoped("GOTOOLCHAIN", "go"+ver)
439471
}
440472
}
441473

442-
return runPlaybackTests(azdDir)
474+
cleanup = func() {
475+
if restoreToolchain != nil {
476+
restoreToolchain()
477+
}
478+
restoreGowork()
479+
}
480+
return azdDir, cleanup, nil
443481
}
444482

445-
// runPlaybackTests discovers test recordings and runs matching functional
446-
// tests in playback mode (AZURE_RECORD_MODE=playback).
447-
func runPlaybackTests(azdDir string) error {
448-
return runPlaybackTestsWithEnv(azdDir, nil)
483+
// testRunOpts configures how runFunctionalTests discovers and executes
484+
// functional tests with recorded HTTP interactions.
485+
type testRunOpts struct {
486+
mode string // "record" or "playback"
487+
filter *string // optional test name filter (substring match)
488+
verbose bool // add -v flag
489+
env []string // additional env vars beyond AZURE_RECORD_MODE
449490
}
450491

451-
// runPlaybackTestsWithEnv is like runPlaybackTests but accepts additional
452-
// environment variables (e.g. CLI_TEST_SKIP_BUILD=true for parallel runs).
453-
func runPlaybackTestsWithEnv(azdDir string, extraEnv []string) error {
492+
// runFunctionalTests discovers playback tests from recordings, applies an
493+
// optional name filter, and runs them via "go test" with the given mode
494+
// and environment variables.
495+
func runFunctionalTests(azdDir string, opts testRunOpts) error {
496+
if opts.mode != "record" && opts.mode != "playback" {
497+
return fmt.Errorf("invalid test mode %q: must be %q or %q", opts.mode, "record", "playback")
498+
}
499+
454500
recordingsDir := filepath.Join(
455501
azdDir, "test", "functional", "testdata", "recordings",
456502
)
457-
names, err := discoverPlaybackTests(recordingsDir)
503+
// In record mode, ignore the stale-recording exclusion list so users can
504+
// re-record those tests via `mage record -filter=<name>`. In playback
505+
// mode, skip them so they don't block preflight.
506+
names, err := discoverPlaybackTests(recordingsDir, opts.mode == "playback")
458507
if err != nil {
459508
return err
460509
}
461510
if len(names) == 0 {
462-
fmt.Println("No recording files found — skipping playback tests.")
511+
fmt.Printf("No recording files found — skipping %s tests.\n", opts.mode)
463512
return nil
464513
}
465514

515+
// Apply optional test filter.
516+
if opts.filter != nil {
517+
var filtered []string
518+
for _, name := range names {
519+
if strings.Contains(name, *opts.filter) || name == *opts.filter {
520+
filtered = append(filtered, name)
521+
}
522+
}
523+
if len(filtered) == 0 {
524+
fmt.Printf("No tests match filter %q in %s mode. Available tests:\n", *opts.filter, opts.mode)
525+
for _, name := range names {
526+
fmt.Printf(" • %s\n", name)
527+
}
528+
return fmt.Errorf("no tests match filter %q", *opts.filter)
529+
}
530+
names = filtered
531+
}
532+
533+
// Build test -run pattern from discovered names.
466534
escaped := make([]string, len(names))
467535
for i, name := range names {
468536
escaped[i] = regexp.QuoteMeta(name)
469537
}
470538
pattern := "^(" + strings.Join(escaped, "|") + ")(/|$)"
471-
fmt.Printf("Running %d tests in playback mode...\n", len(names))
472-
473-
env := append(extraEnv, "AZURE_RECORD_MODE=playback")
474-
return runStreamingWithEnv(
475-
azdDir,
476-
env,
477-
"go", "test", "-run", pattern,
478-
"./test/functional", "-timeout", "30m", "-count=1",
479-
)
539+
540+
fmt.Printf("Running %d test(s) in %s mode...\n", len(names), opts.mode)
541+
if opts.verbose {
542+
for _, name := range names {
543+
fmt.Printf(" • %s\n", name)
544+
}
545+
}
546+
547+
env := []string{"AZURE_RECORD_MODE=" + opts.mode}
548+
env = append(env, opts.env...)
549+
550+
args := []string{"test", "-run", pattern, "./test/functional", "-timeout", "30m", "-count=1"}
551+
if opts.verbose {
552+
args = append(args, "-v")
553+
}
554+
555+
return runStreamingWithEnv(azdDir, env, "go", args...)
480556
}
481557

482558
// excludedPlaybackTests lists tests whose recordings are known to be stale.
483559
// These are excluded from automatic playback so they don't block preflight.
484-
// Re-record the test to remove it from this list.
485-
var excludedPlaybackTests = map[string]string{}
560+
// Re-record the test (mage record -filter=<name>) to remove it from this list.
561+
// Re-recording requires access to the TME subscription — see CONTRIBUTING.md.
562+
var excludedPlaybackTests = map[string]string{
563+
"Test_CLI_VsServer": "stale recording; re-record requires TME access (#7780)",
564+
"Test_CLI_Deploy_SlotDeployment": "stale recording; re-record requires TME access (#7780)",
565+
"Test_CLI_Up_Down_ContainerAppJob": "stale recording; re-record requires TME access (#7014)",
566+
}
486567

487568
// discoverPlaybackTests scans the recordings directory for .yaml files and
488-
// subdirectories, returning unique top-level Go test function names.
489-
func discoverPlaybackTests(recordingsDir string) ([]string, error) {
569+
// subdirectories, returning unique top-level Go test function names. When
570+
// applyExclusions is true, tests in excludedPlaybackTests are filtered out
571+
// (used in playback mode to avoid blocking preflight on known-stale
572+
// recordings). In record mode, callers pass false so excluded tests can
573+
// be re-recorded via `mage record -filter=<name>`.
574+
func discoverPlaybackTests(recordingsDir string, applyExclusions bool) ([]string, error) {
490575
entries, err := os.ReadDir(recordingsDir)
491576
if err != nil {
492577
if errors.Is(err, fs.ErrNotExist) {
@@ -519,9 +604,13 @@ func discoverPlaybackTests(recordingsDir string) ([]string, error) {
519604
seen[cassette] = true
520605
}
521606

522-
// Remove tests with known stale recordings.
523-
for name := range excludedPlaybackTests {
524-
delete(seen, name)
607+
// Remove tests with known stale recordings, but only when requested
608+
// (i.e. in playback mode). Record mode needs to see all tests so users
609+
// can re-record the excluded ones via `mage record -filter=<name>`.
610+
if applyExclusions {
611+
for name := range excludedPlaybackTests {
612+
delete(seen, name)
613+
}
525614
}
526615

527616
if len(seen) == 0 {

0 commit comments

Comments
 (0)