WebSphereLibertyDump failed with exit code 137

[tam512]

Test WebSphereLibertyDump on Power OCP 4.22 on Liberty 26.0.0.7 InstantOn app which was built on UBI10, and it fails to take thread dump with the following error:

Encountered error while running command: [/bin/sh -c mkdir -p /serviceability/daytrader10-instanton/dt10mq-inston-j21-0/serverDumps && server dump --archive=/serviceability/daytrader10-instanton/dt10mq-inston-j21-0/serverDumps/dump_2026.07.01_23.05.17_utc.zip --include=thread,] ; Stderr: ; Error: command terminated with exit code 137

This is WebSphereLibertyDump yaml used to take thread dump

kind: WebSphereLibertyDump
apiVersion: liberty.websphere.ibm.com/v1
metadata:
  name: websphereliberty-dump-sample
  namespace: daytrader10-instanton
spec:
  include:
    - thread
  podName: dt10mq-inston-j21-0

This is a regression because this worked on Liberty 26.0.0.6 with UBI9 app image

[halim-lee]

@tam512 Could I have access to the sample apps (both UBI9 and UBI10)? My apps built on UBI10 and UBI10-minimal did not run into this issue. Is this P particular?

[tam512]

Is yours InstantOn app image? It works ok on zLinux cluster with app without InstantOn. Yes I can you our app images

[tam512]

WebSphereLibertyDump also works ok on x86 OCP cluster with UBI10 InstantOn app image

[abests]

Had a former teammate dealing with this exact exit code 137 on the Liberty UBI10 InstantOn stack during diagnostic dumps.
The root cause is WebSphereLibertyDump triggering an intensive JVM heap/thread-dump sequence which spikes non-heap memory. Under tight cgroup limits, EKS/Fargate has zero tolerance for transient RSS spikes, leading straight to an OOMKill.

We resolved this by enforcing proper memory headroom via LimitRange and VPA:

kind: WebSphereLibertyDump
apiVersion: liberty.websphere.ibm.com/v1
metadata:
  name: websphereliberty-dump-sample
  namespace: daytrader10-instanton
spec:
  include:
    - thread
  podName: dt10mq-inston-j21-0
---
apiVersion: v1
kind: LimitRange
metadata:
  name: daytrader10-instanton-memory-defaults
  namespace: daytrader10-instanton
spec:
  limits:
    - type: Container
      defaultRequest:
        memory: 1Gi
      default:
        memory: 2Gi
      min:
        memory: 512Mi
      max:
        memory: 4Gi
---
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: dt10mq-inston-j21-0-vpa
  namespace: daytrader10-instanton
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: dt10mq-inston-j21-0
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
      - containerName: "*"
        controlledResources:
          - memory
        minAllowed:
          memory: 1Gi
        maxAllowed:
          memory: 4Gi

Highly recommend using the VPA in auto mode to handle the transient peak during the dump without permanently over-provisioning your tasks.

---

P.S. Since debugging these dumps often involves feeding manifests/profiles to LLMs, I open-sourced a local WASM sanitizer we built to scrub AWS keys/secrets before auditing. Dropping it here in case it saves anyone else from a SecOps headache: https://github.com/abests/ghost-sanitizer-js