Skip to content

Commit 47e6345

Browse files
parulbajaj01parulbajaj01
authored
Add MPI operator setup instructions on GKE A4X and A4X Max (#5873)
1 parent dab7e71 commit 47e6345

2 files changed

Lines changed: 156 additions & 0 deletions

File tree

‎examples/gke-a4x-max-bm/README.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,84 @@ This section describes how to run [NCCL/gIB](https://docs.cloud.google.com/ai-hy
120120
kubectl logs $(kubectl get pods -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep nccl-all-worker-0-0)
121121
```
122122

123+
### Install and Run MPI Operator on GKE Cluster
124+
125+
The Kubeflow MPI Operator manages distributed MPI workloads on GKE.
126+
127+
1. **Deploy MPI Operator (v0.8.0):**
128+
129+
* **Automated (During Cluster Creation via Blueprint YAML):** Include the MPI Operator manifest in `apply_manifests` under `kubectl-apply` in your blueprint YAML (`gke-a4x-max-bm.yaml`):
130+
131+
```yaml
132+
- id: kubectl-apply
133+
source: modules/management/kubectl-apply
134+
use: [a4x-max-cluster]
135+
settings:
136+
apply_manifests:
137+
- name: mpi-operator
138+
source: https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.8.0/deploy/v2beta1/mpi-operator.yaml
139+
```
140+
141+
* **Manual (After Cluster Deployment via `kubectl`):** Once the cluster is deployed, run the following command against your cluster:
142+
143+
```bash
144+
kubectl apply --server-side -f https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.8.0/deploy/v2beta1/mpi-operator.yaml
145+
```
146+
147+
2. **Verify Installation:**
148+
149+
```bash
150+
kubectl get crd | grep mpijob
151+
kubectl get pods -n mpi-operator
152+
```
153+
154+
3. **Run a Sample MPIJob Test:**
155+
Create a test manifest `sample-mpijob.yaml`:
156+
157+
```yaml
158+
apiVersion: kubeflow.org/v2beta1
159+
kind: MPIJob
160+
metadata:
161+
name: sample-mpi-job
162+
namespace: default
163+
spec:
164+
slotsPerWorker: 1
165+
runPolicy:
166+
cleanPodPolicy: Running
167+
mpiReplicaSpecs:
168+
Launcher:
169+
replicas: 1
170+
template:
171+
spec:
172+
containers:
173+
- name: mpi-launcher
174+
image: mpioperator/mpi-pi:v0.8.0-openmpi
175+
command:
176+
- mpirun
177+
- --allow-run-as-root
178+
- -n
179+
- "2"
180+
- --hostfile
181+
- /etc/mpi/hostfile
182+
- echo
183+
- "Hello World from MPI worker!"
184+
Worker:
185+
replicas: 2
186+
template:
187+
spec:
188+
containers:
189+
- name: mpi-worker
190+
image: mpioperator/mpi-pi:v0.8.0-openmpi
191+
```
192+
193+
Submit the job and inspect launcher logs:
194+
195+
```bash
196+
kubectl apply -f sample-mpijob.yaml
197+
kubectl logs -l training.kubeflow.org/job-role=launcher
198+
kubectl delete -f sample-mpijob.yaml
199+
```
200+
123201
### Clean up resources created by Cluster Toolkit
124202

125203
To avoid recurring charges for the resources used, clean up the resources provisioned by Cluster Toolkit, including the VPC networks and GKE cluster:

‎examples/gke-a4x/README.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,81 @@ This example provides the configuration to deploy a GKE cluster with A4X machine
55
Refer to [Create an AI-optimized GKE cluster with default configuration](https://cloud.google.com/ai-hypercomputer/docs/create/gke-ai-hypercompute#use-cluster-toolkit) for instructions on creating the GKE-A4X cluster.
66

77
Refer to [Deploy and run NCCL test with Topology Aware Scheduling (TAS)](https://cloud.google.com/ai-hypercomputer/docs/create/gke-ai-hypercompute#deploy-run-nccl-tas-test) for instructions on running a NCCL test on the GKE-A4X cluster.
8+
9+
## Install and Run MPI Operator on GKE Cluster
10+
11+
The Kubeflow MPI Operator manages distributed MPI workloads on GKE.
12+
13+
1. **Deploy MPI Operator (v0.8.0):**
14+
15+
* **Automated (During Cluster Creation via Blueprint YAML):** Include the MPI Operator manifest in `apply_manifests` under `kubectl-apply` in your blueprint YAML (`gke-a4x.yaml`):
16+
17+
```yaml
18+
- id: kubectl-apply
19+
source: modules/management/kubectl-apply
20+
use: [a4x-cluster]
21+
settings:
22+
apply_manifests:
23+
- name: mpi-operator
24+
source: https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.8.0/deploy/v2beta1/mpi-operator.yaml
25+
```
26+
27+
* **Manual (After Cluster Deployment via `kubectl`):** Once the cluster is deployed, run the following command against your cluster:
28+
29+
```bash
30+
kubectl apply --server-side -f https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.8.0/deploy/v2beta1/mpi-operator.yaml
31+
```
32+
33+
2. **Verify Installation:**
34+
35+
```bash
36+
kubectl get crd | grep mpijob
37+
kubectl get pods -n mpi-operator
38+
```
39+
40+
3. **Run a Sample MPIJob Test:**
41+
Create a test manifest `sample-mpijob.yaml`:
42+
43+
```yaml
44+
apiVersion: kubeflow.org/v2beta1
45+
kind: MPIJob
46+
metadata:
47+
name: sample-mpi-job
48+
namespace: default
49+
spec:
50+
slotsPerWorker: 1
51+
runPolicy:
52+
cleanPodPolicy: Running
53+
mpiReplicaSpecs:
54+
Launcher:
55+
replicas: 1
56+
template:
57+
spec:
58+
containers:
59+
- name: mpi-launcher
60+
image: mpioperator/mpi-pi:v0.8.0-openmpi
61+
command:
62+
- mpirun
63+
- --allow-run-as-root
64+
- -n
65+
- "2"
66+
- --hostfile
67+
- /etc/mpi/hostfile
68+
- echo
69+
- "Hello World from MPI worker!"
70+
Worker:
71+
replicas: 2
72+
template:
73+
spec:
74+
containers:
75+
- name: mpi-worker
76+
image: mpioperator/mpi-pi:v0.8.0-openmpi
77+
```
78+
79+
Submit the job and inspect launcher logs:
80+
81+
```bash
82+
kubectl apply -f sample-mpijob.yaml
83+
kubectl logs -l training.kubeflow.org/job-role=launcher
84+
kubectl delete -f sample-mpijob.yaml
85+
```

0 commit comments

Comments
 (0)