Skip to content

Commit e26dd09

Browse files
committed
fix docstring
1 parent 718efc0 commit e26dd09

1 file changed

Lines changed: 12 additions & 12 deletions

File tree

dags/tpu_observability/jobset_ttr_pod_oom.py

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -149,20 +149,20 @@ def pick_random_pod(active_pods: List[str]) -> str:
149149
### JobSet Time-To-Recover (TTR) Test Using Random Pod OOM Injection
150150
### Description
151151
This DAG verifies that JobSet can recover from a single pod failure caused by
152-
an Out-Of-Memory (OOM) event. It launches a JobSet with memory limits,
153-
injects a memory-intensive workload into a running pod, and uses a sensor
154-
to confirm that the JobSet controller triggers a recovery and reports the
155-
recovery duration.
152+
an Out-Of-Memory (OOM) event. It launches a JobSet, injects a memory-intensive
153+
Python stressor into a running pod, and uses a sensor to confirm that the
154+
JobSet controller triggers a recovery and reports the recovery duration (TTR).
156155
### Prerequisites
157-
This test requires an existing cluster and a container image containing
158-
the `oomkill.py` script to run.
156+
This test requires an existing cluster and the ability to execute commands
157+
within the pod via `kubectl exec`.
159158
### Procedures
160-
First, the node pool is created. A JobSet YAML with specific memory constraints
161-
is then launched on the cluster and given time for all pods to reach a
162-
`Running` state. After this, a random pod is selected and an OOM event is
163-
triggered via `kubectl exec` to interrupt the JobSet (expecting Exit Code 137).
164-
A sensor is finally run which will poll Cloud Monitoring to detect that the
165-
JobSet Time-To-Recover (TTR) metric has been updated, resulting in a success,
159+
First, the node pool is created. A JobSet YAML is then launched on the cluster
160+
and given time for all pods to reach a `Running` state. After stabilization,
161+
a random pod is selected and an OOM event is triggered via `kubectl exec`
162+
using a Python-based memory allocator that forces physical RAM commitment
163+
until the process is terminated (expecting Exit Code 137). A sensor is
164+
finally run which will poll Cloud Monitoring to detect that the JobSet
165+
Time-To-Recover (TTR) metric has been updated, resulting in a success,
166166
or timeout, and fail.
167167
""",
168168
) as dag:

0 commit comments

Comments
 (0)