Skip to content

Commit 33a8f3d

Browse files
committed
chaos: log cgroup memory limit and trim aarch64 heap to stop OOMKills
Measured on matching-arch hosts: this workload floats at ~2.6-2.75GB RSS regardless of MALLOC_CHECK_/allocator tuning, ruling that out as the OOM cause. Trim -Xmx to 1536m on aarch64 for real headroom, and log the container's actual cgroup memory limit so future OOMs are diagnosable from the job log directly.
1 parent cfe1245 commit 33a8f3d

1 file changed

Lines changed: 18 additions & 1 deletion

File tree

utils/run-chaos-harness.sh

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -182,6 +182,23 @@ case $ALLOCATOR in
182182
esac
183183
echo "LD_PRELOAD=${LD_PRELOAD:-}"
184184

185+
# Log the actual container memory ceiling (if any) so OOMKilled failures are
186+
# diagnosable from the CI job log directly, instead of having to infer it
187+
# from RSS measurements on a separate repro host.
188+
CGROUP_MEM_LIMIT=$(cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null || cat /sys/fs/cgroup/memory.max 2>/dev/null || echo "unknown")
189+
echo "cgroup memory limit: ${CGROUP_MEM_LIMIT}"
190+
191+
# Measured on a matching-arch host: this workload's RSS floats around
192+
# 2.6-2.75GB at -Xmx2g regardless of allocator/malloc tuning (confirmed the
193+
# aarch64 CI OOMs are not caused by MALLOC_CHECK_'s per-chunk overhead -
194+
# RSS was the same or higher with it disabled), meaning the container's
195+
# actual memory ceiling sits at or below that. Trim the heap on aarch64 to
196+
# give real headroom instead of re-tuning allocator env vars again.
197+
HEAP_MB=2048
198+
if [ "$(uname -m)" = "aarch64" ] || [ "$(uname -m)" = "arm64" ]; then
199+
HEAP_MB=1536
200+
fi
201+
185202
HS_ERR="${CHAOS_ERROR_FILE:-${WORK_DIR}/hs_err.log}"
186203
rm -f "${HS_ERR}"
187204

@@ -198,7 +215,7 @@ java -javaagent:${PATCHED_AGENT} \
198215
-Ddd.profiling.ddprof.nativemem.enabled=true \
199216
-Ddd.env=java-profiler-stability \
200217
-Ddd.service=java-profiler-chaos \
201-
-Xmx2g -Xms2g \
218+
-Xmx${HEAP_MB}m -Xms${HEAP_MB}m \
202219
-XX:MaxMetaspaceSize=384m \
203220
-XX:NativeMemoryTracking=summary \
204221
-XX:ErrorFile=${HS_ERR} \

0 commit comments

Comments
 (0)