Skip to content

Use larger Namespace runner for build-cli-pr job#614

Merged
dahlia merged 3 commits into
fedify-dev:mainfrom
sij411:fix/build-cli-pr-oom
Mar 12, 2026
Merged

Use larger Namespace runner for build-cli-pr job#614
dahlia merged 3 commits into
fedify-dev:mainfrom
sij411:fix/build-cli-pr-oom

Conversation

@sij411

@sij411 sij411 commented Mar 11, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Switch build-cli-pr to namespace-profile-linux-amd64-16gb runner to fix intermittent OOM kills during parallel deno compile

Context

The job runs 5 deno compile processes in parallel via pack.ts, each peaking at ~4-5GB. On ubuntu-latest (~7GB RAM) this intermittently causes OOM (exit 137).

Tested locally with Docker memory limits:

Concurrency Memory Result
5 6GB OOM
2 7GB OOM
1 7GB Passed (~14min)

16GB runner provides sufficient headroom for all 5 parallel compiles.

Test plan

  • Verify build-cli-pr job passes on the new runner
  • Monitor for OOM flakes over the next few PRs

Closes #605

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Note

Gemini is unable to generate a summary for this pull request due to the file types involved not being currently supported.

@codecov

codecov Bot commented Mar 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@dahlia dahlia left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we update the build-cli job as well?

@sij411

sij411 commented Mar 11, 2026

Copy link
Copy Markdown
Contributor Author

Shouldn't we update the build-cli job as well?

I thought that was a future plan. I'll change it.

@sij411 sij411 force-pushed the fix/build-cli-pr-oom branch from 4118f57 to 8d8b7a4 Compare March 11, 2026 11:52
Comment thread packages/cli/scripts/pack.ts Outdated
@sij411 sij411 force-pushed the fix/build-cli-pr-oom branch from 3c9d9f1 to 6605297 Compare March 12, 2026 05:47
sij411 and others added 3 commits March 12, 2026 15:52
The build-cli-pr job intermittently failed with "The runner has received
a shutdown signal" (OOM kill, exit code 137) because pack.ts runs 5
deno compile processes in parallel, each peaking at ~4-5GB memory,
which exceeds the ~7GB available on ubuntu-latest runners.

Reproduced locally with Docker memory limits:

  - Concurrency 5, 6GB: OOM (exit 137)
  - Concurrency 2, 7GB: OOM (exit 137)
  - Concurrency 1, 7GB: passed (~14min)

The failure is non-deterministic because OOM depends on whether peak
memory phases of the 5 compile processes overlap in time.

Switch build-cli-pr to a Namespace runner with 16GB RAM, which provides
sufficient headroom for all 5 parallel compiles.

Resolves fedify-dev#605

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Same OOM risk as build-cli-pr since it runs the same pack.ts script.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Even with the 16GB Namespace runner, 5 parallel deno compile processes
still caused OOM (exit 137). Each compile peaks at ~4-5GB, so 5 in
parallel can exceed 16GB.

Add a concurrency pool that limits parallel compiles. The default is
all targets (preserving original behavior for local development), but
can be overridden via the CONCURRENCY env var. CI workflows set
CONCURRENCY=3 to keep peak memory at ~12-15GB.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@sij411 sij411 force-pushed the fix/build-cli-pr-oom branch from 6605297 to d3f1352 Compare March 12, 2026 06:52

@dahlia dahlia left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything looks great. Thanks!

@dahlia dahlia merged commit e79f888 into fedify-dev:main Mar 12, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

build-cli-pr job intermittently OOM-killed due to parallel deno compile

2 participants