Fast Deployment XA0137 (run-as 'couldn't stat') races install on the PRIMARY user (user 0) ~daily in CI

[simonrozsival]

Summary

The Fast Deployment install-vs-run-as race (#7821) is firing in CI on the primary user (user 0), which the current EnsureUserIsRunning mitigation deliberately skips. It recurs ~daily on the public dotnet-android pipeline, always in the same job/step, and is not machine-specific.

.../Xamarin.Android.Common.Debugging.targets(333,5): error XA0137:
  The 'run-as' command failed with 'run-as: couldn't stat
  /data/user/0/Mono.Android.NET_Tests: No such file or directory'.
... Fast Deployment is not currently supported on this device.

Why it's easy to miss

The failure is emitted by the -t:Install step, which runs with continueOnError: true. As a result it does not surface as:

• a failed test result (scanned 94 failed builds' test messages → 0 hits), or
• a timeline issue on the job (→ 0 hits).

It only appears in the raw install-step log. An exhaustive grep of all logs across 105 public builds (2026‑06‑26 → 06‑29, ~46k log files) found it. The build step is marked succeededWithIssues, but its parent job macOS > Tests > APKs 1 fails.

Occurrences (dnceng-public dotnet-android, definition 333)

6 distinct occurrences in 4 days, each on an unrelated PR and a different agent:

Build	PR	Date (UTC)	Agent (workerName)	Job → Step
1482888	#11730	2026-06-26	Azure Pipelines 3	macOS > Tests > APKs 1 → build Mono.Android.NET_Tests-Debug
1482927	#11754	2026-06-26	Azure Pipelines 54	macOS > Tests > APKs 1 → build Mono.Android.NET_Tests-Debug
1483263	#11758	2026-06-26	Azure Pipelines 41	macOS > Tests > APKs 1 → build Mono.Android.NET_Tests-Debug
1484013	#11765	2026-06-27	Azure Pipelines 40	macOS > Tests > APKs 1 → build Mono.Android.NET_Tests-Debug
1484638	#11778	2026-06-28	Azure Pipelines 6	macOS > Tests > APKs 1 → build Mono.Android.NET_Tests-Debug
1484387	#11670	2026-06-29	Azure Pipelines 36	macOS > Tests > APKs 1 → build Mono.Android.NET_Tests-Debug

No traces were found in older builds — they very likely succeeded on auto-retry and the install-step logs aged out of retention.

Analysis

Not machine-specific. Six occurrences on six distinct agents (3, 6, 36, 40, 41, 54) ⇒ this is a systemic timing race, not a bad node.

Invariant across every hit:

• Same job/step: macOS > Tests > APKs 1 → build Mono.Android.NET_Tests-Debug (-t:Install).
• Mono.Android.NET_Tests-Debug uses EmbedAssembliesIntoApk=false ⇒ the Fast Deployment run-as code path.
• Always user 0 (primary) and package Mono.Android.NET_Tests.
• Recurs ~daily (3× Jun 26, then 27/28/29).

Root cause

This is the install-vs-run-as race from #7821: pm install registers the package but the per-user data dir /data/user/0/<pkg> is not yet stat-able through run-as when Fast Dev issues its first run-as <pkg> "pwd" query (FastDeploy.CheckAppInstalledAndDebuggable), so run-as returns couldn't stat … No such file or directory and RaiseRunAsError emits XA0137.

EnsureUserIsRunning (added for #11322) fixes this for secondary users via am start-user -w N, but explicitly skips user 0:

// FastDeploy.cs
if (userId.Length == 0 || (int.TryParse (userId, out var id) && id == 0)) {
    return;   // "The primary user (id 0) never requires this step"
}

The data above shows that assumption has a hole: the user-0 install of Mono.Android.NET_Tests-Debug hits the same race ~daily.

Proposed fix

Make the first run-as query tolerant of the t=0 race for all users, e.g. a bounded retry/poll in CheckAppInstalledAndDebuggable when the result contains couldn't stat … No such file or directory, before raising XA0137. Optionally, stop skipping user 0 in EnsureUserIsRunning (a am start-user -w 0 / wait-for-data-dir is a cheap no-op when already running).

Affected files

• src/Xamarin.Android.Build.Debugging.Tasks/Tasks/FastDeploy.cs (CheckAppInstalledAndDebuggable, EnsureUserIsRunning, RaiseRunAsError)

Related

• When using FastDeployment get error message XA0136: The 'run-as' command failed with 'run-as: Package 'packagename' has corrupt installation #7821 — original install-vs-run-as race report
• Investigate ApplicationRunsWithDebuggerAndBreaks flakiness: Fast Deployment + guest user (XA0137) + test-code bugs (follow-up to #11320) #11322 — XA0137 + guest-user flakiness (secondary-user fix via EnsureUserIsRunning)