[NativeAOT] Print OOM message before Abort() on Linux

[Copilot]

On Linux, NativeAOT processes terminating due to OutOfMemoryException (e.g. with DOTNET_GCHeapHardLimit set) printed only Aborted with no diagnostic context.

Root cause

RuntimeExceptionHelpers.FailFast detects the preallocated OOM exception via minimalFailFast = (exception == PreallocatedOutOfMemoryException.Instance) and skips all stderr output to avoid heap allocations — leaving the user with no indication of why the process died.

Fix

In the minimalFailFast path, print a hardcoded OOM message to stderr before calling Abort(). The write is wrapped in try/catch {} so a secondary allocation failure silently falls through to the existing abort path.

Before: Aborted
After: Out of memory.

Test

Added a new OomHandling smoke test in src/tests/baseservices/exceptions/OutOfMemoryException/. The test spawns itself as a subprocess with DOTNET_GCHeapHardLimit=20000000 (32 MB) set, waits for the subprocess to run out of memory, and verifies that OutOfMemoryException appears in stderr. This covers both the preallocated OOM path (the fix) and the existing unhandled-exception path. The test is skipped on mobile and browser platforms that do not support process spawning.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[NativeAOT] Out of memory reporting on Linux</issue_title>
<issue_description>### Repro

var l = new List<object>();
for (; ; ) l.Add(new object());

Run the native aot compiled binary with with export DOTNET_GCHeapHardLimit=2000000 set

Actual result

Aborted

Expected result

Out of memory.

(Reported by partner team.)</issue_description>

Comments on the Issue (you are @copilot in this section)

@ Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas See info in [area-owners.md](https://github.com/dotnet/runtime/blob/main/docs/area-owners.md) if you want to be subscribed.

Issue Details

---

Repro

var l = new List<object>();
for (; ; ) l.Add(new object());

Run the native aot compiled binary with with export DOTNET_GCHeapHardLimit=2000000 set

Actual result

Aborted

Expected result

Process is terminating due to OutOfMemoryException

(Reported by partner team.)

Author:	jkotas
Assignees:	-
Labels:	area-NativeAOT-coreclr
Milestone:	8.0.0

• Fixes [NativeAOT] Out of memory reporting on Linux #82337

---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

[agocke]

Please add a unit test to src/tests

[agocke]

@copilot

[Copilot]

Please add a unit test to src/tests

Added a new OomHandling smoke test in src/tests/nativeaot/SmokeTests/OomHandling/ (commit 3ef2a6d). The test spawns itself as a subprocess with DOTNET_GCHeapHardLimit=20000000 (20 MB) set, waits for it to OOM, and verifies that OutOfMemoryException appears in stderr — covering both the preallocated OOM path (my fix) and the existing unhandled-exception path. The test is skipped on mobile and browser platforms that do not support process spawning.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.