@@ -149,20 +149,20 @@ def pick_random_pod(active_pods: List[str]) -> str:
149149 ### JobSet Time-To-Recover (TTR) Test Using Random Pod OOM Injection
150150 ### Description
151151 This DAG verifies that JobSet can recover from a single pod failure caused by
152- an Out-Of-Memory (OOM) event. It launches a JobSet with memory limits,
153- injects a memory-intensive workload into a running pod, and uses a sensor
154- to confirm that the JobSet controller triggers a recovery and reports the
155- recovery duration.
152+ an Out-Of-Memory (OOM) event. It launches a JobSet, injects a memory-intensive
153+ Python stressor into a running pod, and uses a sensor to confirm that the
154+ JobSet controller triggers a recovery and reports the recovery duration (TTR).
156155 ### Prerequisites
157- This test requires an existing cluster and a container image containing
158- the `oomkill.py` script to run .
156+ This test requires an existing cluster and the ability to execute commands
157+ within the pod via `kubectl exec` .
159158 ### Procedures
160- First, the node pool is created. A JobSet YAML with specific memory constraints
161- is then launched on the cluster and given time for all pods to reach a
162- `Running` state. After this, a random pod is selected and an OOM event is
163- triggered via `kubectl exec` to interrupt the JobSet (expecting Exit Code 137).
164- A sensor is finally run which will poll Cloud Monitoring to detect that the
165- JobSet Time-To-Recover (TTR) metric has been updated, resulting in a success,
159+ First, the node pool is created. A JobSet YAML is then launched on the cluster
160+ and given time for all pods to reach a `Running` state. After stabilization,
161+ a random pod is selected and an OOM event is triggered via `kubectl exec`
162+ using a Python-based memory allocator that forces physical RAM commitment
163+ until the process is terminated (expecting Exit Code 137). A sensor is
164+ finally run which will poll Cloud Monitoring to detect that the JobSet
165+ Time-To-Recover (TTR) metric has been updated, resulting in a success,
166166 or timeout, and fail.
167167 """ ,
168168) as dag :
0 commit comments