Skip to content

Commit ad38f5e

Browse files
committed
docs: add known issues
1 parent 9455431 commit ad38f5e

1 file changed

Lines changed: 49 additions & 1 deletion

File tree

  • examples/science/af3-slurm/examples/simple_ipynb_launcher

examples/science/af3-slurm/examples/simple_ipynb_launcher/README.md

Lines changed: 49 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,12 @@ Please note that the launcher needs 2 specific setup steps:
1111
## Usage Guide
1212
For usage of the Ipynb Launcher consult the [Step-by-Step Instructions](./Ipynb.md)
1313

14-
## Known Limitations
14+
## Known Issues
1515
You may encounter the following problems while using the notebook.
1616

1717
### Warning during dependency installation
18+
19+
**Description**:
1820
You may encounter the following warning during dependency installation:
1921

2022
```text
@@ -25,3 +27,49 @@ google-cloud-bigtable 1.7.3 requires grpc-google-iam-v1<0.13dev,>=0.12.3, but yo
2527
**Resolution**: This warning can be **safely ignored**.
2628

2729
The version mismatch does not impact the functionality required by this project. The `google-cloud-bigtable` package is not used in any critical code path, and no issues have been observed during execution.
30+
31+
### Resource Unavailability (Out of Capacity)
32+
33+
**Description**:
34+
Your job is queued but not executed because the cloud provider (e.g., GCP) can't provision the required compute nodes. This is often due to a temporary lack of available resources in the chosen region or zone.
35+
36+
**Resolution**:
37+
- Check the **Controller Logs**:
38+
Log on to the controller machine for your cluster to see detailed error messages like
39+
`"GCP Error: Zone does not currently have sufficient capacity for resources"`.
40+
- **Wait and Re-check**:
41+
For temporary resource shortages, wait **5–10 minutes** and then re-run the "check job status" cell in your Jupyter notebook. Resources might become available and allow your job to proceed.
42+
43+
### Node Creation In Progress
44+
45+
**Description**:
46+
After your job is submitted, it takes time for the cluster to spin up new nodes. You might see a `NODE_FAIL` status if nodes are still being created and are not yet ready.
47+
48+
**Resolution**:
49+
- Monitor the **Controller Logs** for status updates.
50+
- **Wait 5–10 minutes**, then recheck your job status to see if nodes have become available.
51+
52+
---
53+
54+
### Runtime-Related Issues
55+
56+
These issues occur once your job attempts to run on the allocated nodes.
57+
58+
### Input File Format Error
59+
60+
**Description**:
61+
Your job might fail immediately if the input file format is incorrect or if the file is corrupted.
62+
63+
**Resolution**:
64+
- Review the **Job Logs** to identify input file issues.
65+
- Ensure the input file meets the expected format and isn't corrupted.
66+
67+
### Out of Memory (OOM) Issue
68+
69+
**Description**:
70+
Your job may start but fail during execution because it requires more memory than the allocated node can provide. This can occur in both the **datapipeline** and **inference** stages.
71+
72+
**Resolution**:
73+
- Check the **Job Logs** for OOM errors or memory-related failures.
74+
- Modify your code or input data to reduce memory usage.
75+
- Consider requesting **larger node sizes** with higher memory capacity.

0 commit comments

Comments
 (0)