Skip to content

Possible k8s OOM Kill prevention pill 2 - rlimit#1370

Draft
taylordowns2000 wants to merge 4 commits intomainfrom
memory-concept-2
Draft

Possible k8s OOM Kill prevention pill 2 - rlimit#1370
taylordowns2000 wants to merge 4 commits intomainfrom
memory-concept-2

Conversation

@taylordowns2000
Copy link
Copy Markdown
Member

@taylordowns2000 taylordowns2000 commented Apr 13, 2026

Background: It took me about 20 seconds to crash a staging worker in Kubernetes:

image image

The above run will show up as "lost" in the next 30 minutes.


This PR uses prlimit to set RLIMIT_AS on each forked child process, capping virtual address space so a runaway
run crashes itself instead of OOM-killing the pod.

It's opt-in by detection: active when prlimit (from util-linux) is available on Linux; no-op on macOS / local dev, and it adds util-linux to the worker Docker image so it's available

Testing on staging

  1. Deploy this branch to a worker connected to app.staging.openfn.org
  2. Create a workflow with a job that spikes memory beyond the configured limit, e.g.:
  fn(state => {
    const arr = [];
    while (true) { arr.push(new Array(1e6).fill('x')); }
    return state;
  });
  1. Run the workflow and confirm:
  • The run fails with an OOM error (not a pod restart)
  • Other concurrent runs on the same worker are unaffected
  • The worker recovers and picks up new runs normally
  1. Check worker logs for cgroup memory enforcement enabled at startup and killed by SIGKILL (probable OOM) on the failing run
  2. Verify no leftover openfn-worker-* directories under the cgroup root after the run completes

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):

  • I have used Claude Code
  • I have used another model
  • I have not used AI

You can read more details in our
Responsible AI Policy

Release branch checklist

Delete this section if this is not a release PR.

If this IS a release branch:

  • Run pnpm changeset version from root to bump versions
  • Run pnpm install
  • Commit the new version numbers
  • Run pnpm changeset tag to generate tags
  • Push tags git push --tags

Tags may need updating if commits come in after the tags are first generated.

@github-project-automation github-project-automation bot moved this to New Issues in Core Apr 13, 2026
@taylordowns2000 taylordowns2000 changed the title cgroup Possible k8s OOM Kill prevention via cgroup - pill 2 Apr 13, 2026
@taylordowns2000 taylordowns2000 changed the title Possible k8s OOM Kill prevention via cgroup - pill 2 Possible k8s OOM Kill prevention pill 2 - cgroup Apr 13, 2026
@taylordowns2000 taylordowns2000 changed the title Possible k8s OOM Kill prevention pill 2 - cgroup Possible k8s OOM Kill prevention pill 2 - rlimit Apr 13, 2026
@josephjclark
Copy link
Copy Markdown
Collaborator

Gosh there's a lot of stuff here, and I have no idea what any of it does. I'll take a close look at it (probably tomorrow)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: New Issues

Development

Successfully merging this pull request may close these issues.

2 participants