You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: examples/science/af3-slurm/examples/simple_ipynb_launcher/README.md
+49-1Lines changed: 49 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,10 +11,12 @@ Please note that the launcher needs 2 specific setup steps:
11
11
## Usage Guide
12
12
For usage of the Ipynb Launcher consult the [Step-by-Step Instructions](./Ipynb.md)
13
13
14
-
## Known Limitations
14
+
## Known Issues
15
15
You may encounter the following problems while using the notebook.
16
16
17
17
### Warning during dependency installation
18
+
19
+
**Description**:
18
20
You may encounter the following warning during dependency installation:
19
21
20
22
```text
@@ -25,3 +27,49 @@ google-cloud-bigtable 1.7.3 requires grpc-google-iam-v1<0.13dev,>=0.12.3, but yo
25
27
**Resolution**: This warning can be **safely ignored**.
26
28
27
29
The version mismatch does not impact the functionality required by this project. The `google-cloud-bigtable` package is not used in any critical code path, and no issues have been observed during execution.
30
+
31
+
### Resource Unavailability (Out of Capacity)
32
+
33
+
**Description**:
34
+
Your job is queued but not executed because the cloud provider (e.g., GCP) can't provision the required compute nodes. This is often due to a temporary lack of available resources in the chosen region or zone.
35
+
36
+
**Resolution**:
37
+
- Check the **Controller Logs**:
38
+
Log on to the controller machine for your cluster to see detailed error messages like
39
+
`"GCP Error: Zone does not currently have sufficient capacity for resources"`.
40
+
-**Wait and Re-check**:
41
+
For temporary resource shortages, wait **5–10 minutes** and then re-run the "check job status" cell in your Jupyter notebook. Resources might become available and allow your job to proceed.
42
+
43
+
### Node Creation In Progress
44
+
45
+
**Description**:
46
+
After your job is submitted, it takes time for the cluster to spin up new nodes. You might see a `NODE_FAIL` status if nodes are still being created and are not yet ready.
47
+
48
+
**Resolution**:
49
+
- Monitor the **Controller Logs** for status updates.
50
+
-**Wait 5–10 minutes**, then recheck your job status to see if nodes have become available.
51
+
52
+
---
53
+
54
+
### Runtime-Related Issues
55
+
56
+
These issues occur once your job attempts to run on the allocated nodes.
57
+
58
+
### Input File Format Error
59
+
60
+
**Description**:
61
+
Your job might fail immediately if the input file format is incorrect or if the file is corrupted.
62
+
63
+
**Resolution**:
64
+
- Review the **Job Logs** to identify input file issues.
65
+
- Ensure the input file meets the expected format and isn't corrupted.
66
+
67
+
### Out of Memory (OOM) Issue
68
+
69
+
**Description**:
70
+
Your job may start but fail during execution because it requires more memory than the allocated node can provide. This can occur in both the **datapipeline** and **inference** stages.
71
+
72
+
**Resolution**:
73
+
- Check the **Job Logs** for OOM errors or memory-related failures.
74
+
- Modify your code or input data to reduce memory usage.
75
+
- Consider requesting **larger node sizes** with higher memory capacity.
0 commit comments