Skip to content

Commit 758501e

Browse files
mesakhcienetmesakhcienet
committed
feat: polish readme
1 parent 2466406 commit 758501e

1 file changed

Lines changed: 63 additions & 121 deletions

File tree

‎examples/science/af3-slurm/examples/simple_ipynb_launcher/Setup-post-cluster-deployment.md‎

Lines changed: 63 additions & 121 deletions
Original file line numberDiff line numberDiff line change
@@ -1,163 +1,100 @@
11
# Setup: Post-Cluster Deployment
2-
This guide explains how to properly set up and launch the Simple IPython Launcher environment **after deploying the `af3-slurm.yaml` cluster**.
32

4-
## Prerequisites
5-
6-
### Set up Jobs Bucket
7-
8-
If you want to use the simple service launcher, you need to create an additional bucket, that should
9-
be located in the region where you stand up your cluster:
10-
11-
```bash
12-
#!/bin/bash
13-
14-
UNIQUE_JOB_BUCKET=<your-bucket>
15-
PROJECT_ID=<your-gcp-project>
16-
REGION=<your-preferred-region>
17-
18-
gcloud storage buckets create gs://${UNIQUE_JOB_BUCKET} \
19-
--project=${PROJECT_ID} \
20-
--default-storage-class=STANDARD --location=${REGION} \
21-
--uniform-bucket-level-access
22-
```
23-
24-
### Teardown SLURM cluster
25-
26-
If you deployed the SLURM cluster without setting up the necessary configurations for the IPython Notebook (such as `af3ipynb_bucket`, `slurm_rest_token_secret_name`, or notebook-specific variables), you must tear down the existing SLURM cluster before proceeding.
27-
28-
This is required because these configurations are only applied at deployment time. Modifying them after the cluster is live will not properly enable the notebook environment or API integration.
3+
This guide explains how to deploy and access the Jupyter Notebook environment for AlphaFold **after the af3-slurm.yaml cluster has already been deployed** with SLURM REST support enabled.
294

30-
Run the following command from the cluster-toolkit root directory to destroy the previously deployed SLURM cluster:
31-
32-
```bash
33-
# Make sure you're in the root of the `cluster-toolkit` directory
34-
cluster-toolkit$
35-
./gcluster destroy af3-slurm --only cluster --auto-approve
36-
```
37-
38-
After tearing down the cluster, you can safely update `af3-slurm-deployment.yaml` with the necessary notebook settings. The next section explains how to make these updates.
39-
40-
## Deploying a Jupyter Notebook for AlphaFold
41-
42-
You can deploy a Jupyter Notebook environment to run AlphaFold step-by-step. This is useful for interactive exploration and making custom modifications to the AlphaFold pipeline.
43-
44-
> **Important:** To enable the Jupyter Notebook deployment, you **must provide a cloud storage bucket**. If no bucket is specified, the notebook environment will not be created.
45-
46-
### Enable the SLURM REST API Server
5+
## Prerequisites
476

48-
Set the `slurm_rest_server_activate` value to `true` in the `af3-slurm-deployment.yaml` file to enable the SLURM REST API server on the controller node of the `af3-slurm.yaml` cluster. This allows the Jupyter Notebook environment to send requests—such as job submissions—to the SLURM scheduler via the REST API.
7+
Before proceeding, ensure the following configuration values were set correctly in your `af3-slurm-deployment.yaml` file **prior to deploying the SLURM cluster**:
498

509
```yaml
5110
slurm_rest_server_activate: true
11+
slurm_rest_token_secret_name: "<your-secret-name>" # Name of your Secret Manager secret
12+
af3ipynb_bucket: "<your-pre-existing-bucket-name>" # Existing Cloud Storage bucket name
5213
```
5314
54-
### Configuring the SLURM REST API Token Secret Manager
55-
56-
Set `slurm_rest_token_secret_name` value in the `af3-slurm-deployment.yaml` with the name of an existing secret in Secret Manager. Alternatively, you can specify a name for a secret that does not yet exist. If the specified secret name is new, this blueprint will automatically create it for you.
57-
58-
> This setting allows you to specify the name of a Google Cloud Secret Manager secret that holds your SLURM authentication token. Using Secret Manager is a secure way to manage sensitive credentials.
59-
60-
```yaml
61-
slurm_rest_token_secret_name: "<your-secret-name>"
62-
```
63-
64-
### Steps to Deploy
65-
66-
1. **Set the notebook storage bucket** in your `af3-slurm-deployment.yaml` file:
67-
68-
```yaml
69-
af3ipynb_bucket: "<your-pre-existing-bucket-name>"
70-
```
71-
72-
> This bucket is used to store notebooks and related files for the Jupyter environment.
73-
74-
2. **Deploy the Jupyter Notebook environment** using the `af3-slurm-ipynb.yaml` blueprint:
15+
> [!WARNING]
16+
> This guide assumes that the SLURM cluster is already deployed with REST API support enabled—i.e., the values above were correctly set before deployment.
17+
> If any of the values above were omitted or misconfigured, you must destroy the cluster, update the deployment configuration, and redeploy. These settings are essential for enabling SLURM REST-based notebook functionality.
18+
> This document focuses exclusively on deploying and accessing the Jupyter Notebook environment **after** the cluster is running and properly configured. It does not cover modifying or redeploying the cluster.
7519
76-
```bash
77-
# Make sure you're in the root of the `cluster-toolkit` directory
78-
cluster-toolkit$ ./gcluster deploy \
79-
-d examples/science/af3-slurm/af3-slurm-deployment.yaml \
80-
examples/science/af3-slurm/examples/simple_ipynb_launcher/af3-slurm-ipynb.yaml \
81-
--auto-approve
82-
```
20+
If you have **not yet deployed** the cluster, or if you need help setting the appropriate variables prior to deployment, refer to [Setup-pre-cluster-deployment.md](./Setup-pre-cluster-deployment.md) guide.
8321
84-
3. **Rebuild the SLURM cluster** using the updated deployment file:
22+
## Deploying the Jupyter Notebook Blueprint
8523
86-
```bash
87-
# From the `cluster-toolkit` root directory
88-
cluster-toolkit$ ./gcluster deploy \
89-
-d examples/science/af3-slurm/af3-slurm-deployment.yaml \
90-
examples/science/af3-slurm/af3-slurm.yaml \
91-
--only cluster \
92-
--auto-approve
93-
```
24+
### 1. Upload the required Notebook to the Cloud Storage Bucket
9425
95-
## Upload Notebook to Bucket
26+
First, access the controller node of your deployed SLURM cluster. Replace placeholders `<controller-node-name>` and `<your-zone>` with your actual node and zone.
9627

97-
To upload the Jupyter notebook to the cloud storage bucket so it can be accessed via JupyterLab:
28+
```bash
29+
gcloud compute ssh <controller-node-name> --zone=<your-zone>
30+
```
9831

99-
1. **SSH into controller node** in the cluster.
32+
On the controller node, run the following command (assuming the `slurm_rest_user` value in `af3-slurm-deployment.yaml` has not changed):
10033

101-
2. **Navigate to the setup directory**:
34+
```bash
35+
cd /home/af3ipynb/ipynb_setup
36+
ansible-playbook ipynb-upload-config.yml
37+
```
10238

103-
```bash
104-
cd /home/af3ipynb/ipynb_setup
105-
```
39+
This step uploads the notebook (`slurm-rest-api-notebook.ipynb`) along with its required scripts and libraries to the bucket defined in the `af3ipynb_bucket` variable in `af3-slurm-deployment.yaml` file.
40+
Make sure that this bucket was created and specified during the SLURM cluster deployment process.
10641

107-
3. Run the Ansible playbook to upload the Jupyter notebook and its required library files to the designated bucket:
42+
### 2. Grant Secret Access to the Notebook's Service Account
10843

109-
```bash
110-
ansible-playbook ipynb-upload-config.yml
111-
```
44+
This setting ensures that the notebook server can successfully retrieve the specified secret by name.
45+
Make sure the service account running the Jupyter Notebook instance (typically the Compute Engine default service account) has permission to access the Secret Manager secret that stores your SLURM REST token.
11246

113-
This playbook will upload `slurm-rest-api-notebook.ipynb` along with its associated scripts and library files to the target bucket (`af3ipynb_bucket`).
114-
Once the upload is complete, you can access the notebook from the JupyterLab interface via:
47+
The following command grants the Compute Engine default service account the `roles/secretmanager.secretAccessor` role, allowing it to access the specified secret in Secret Manager. The attached condition always evaluates to true, ensuring access is consistently granted.
11548

116-
> Cloud Console → Vertex AI → Workbench → Instances
49+
```bash
50+
gcloud secrets add-iam-policy-binding <your-secret-name> \
51+
--member="serviceAccount:$(gcloud projects describe <your-project-id> --format='value(projectNumber)')-compute@developer.gserviceaccount.com" \
52+
--role="roles/secretmanager.secretAccessor" \
53+
--condition="expression=true,title=AlwaysTrue,description=Allow access to Secret Manager"
54+
```
11755

118-
## Granting Access to the Token
56+
You can verify this configuration in the Secret Manager section of the Google Cloud Console.
11957

120-
> Before running this section, please complete the [**Upload Notebook to Bucket**](#upload-notebook-to-bucket) section first.
58+
<img src="adm/secret-manager.png" alt="secret-manager" width="1000">
12159

122-
After the deploying steps are success, you must ensure that the appropriate service account (e.g., the default service account for the Compute Engine instance) has permission to access the secret. This step is crucial because, without access to the secret, the notebook will not be able to authenticate properly or send valid requests to the SLURM REST API.
60+
### 4. Launch the Notebook Environment
12361

124-
If you have not granted access yet, then run the following command, replacing `<your-secret-name>` and `<your-project-id>` with your values:
62+
Deploy the Jupyter Notebook environment using the following command:
12563

126-
```bash
127-
gcloud secrets add-iam-policy-binding <your-secret-name> \
128-
--member="serviceAccount:$(gcloud projects describe <your-project-id> --format='value(projectNumber)')-compute@developer.gserviceaccount.com" \
129-
--role="roles/secretmanager.secretAccessor" \
130-
--condition="expression=true,title=AlwaysTrue,description=Allow access to Secret Manager"
131-
```
64+
```bash
65+
# Move to cluster-toolkit root folder
66+
cd cluster-toolkit
67+
./gcluster deploy -d examples/science/af3-slurm/af3-slurm-deployment.yaml \
68+
examples/science/af3-slurm/examples/simple_ipynb_launcher/af3-slurm-ipynb.yaml --auto-approve
69+
```
13270

133-
This command grants the Compute Engine default service account the `secretAccessor` role with a condition that always evaluates to true, as shown in the image below. You can view this in the `Secret Manager` page in the Google Cloud Console.
71+
This brings up the Jupyter Notebook deployment.
13472

135-
<img src="adm/secret-manager.png" alt="secret-manager" width="1000">
73+
### 5. Access the Notebook via Vertex AI Workbench
13674

137-
This setting allows the notebook server to successfully retrieve the value of the specified secret by its name.
75+
In the Google Cloud Console:
13876

139-
### Verify Access
77+
1. Navigate to `Vertex AI` → `Workbench` → `Instances`
14078

141-
To confirm that the service account has the necessary permissions, from the jupyter notebook workbench, you can open a terminal and run the following command:
79+
2. Open the JupyterLab interface for the newly deployed instance
14280

143-
```bash
144-
gcloud secrets versions access latest --secret=<your-secret-name>
145-
```
81+
3. Locate and open the `slurm-rest-api-notebook.ipynb` file. If you haven't modified the default value of `af3ipynb_bucket_local_mount` in the `af3-slurm-deployment.yaml`, the notebook should be available at `/home/jupyter/alphafold` folder.
14682

147-
If the command succeeds, this confirms that your notebook can securely retrieve the authentication token from Secret Manager as shown in the image below.
83+
### 6. Verify REST Token Access
14884

149-
<img src="adm/rest_api.png" alt="slrum rest api" width="1000">
85+
To verify that Secret Manager access is properly configured, open a terminal within JupyterLab and run the following command:
15086

151-
## Custom configuration
87+
```bash
88+
gcloud secrets versions access latest --secret=<your-secret-name>
89+
```
15290

153-
You can customize settings via blueprint variables before deployment. If modifications are needed later, ensure:
91+
If the command returns the secret value successfully, it confirms that the notebook environment can securely access the SLURM REST API—just as in the image below:
15492

155-
- All required resources are available
156-
- Configuration changes within the notebook are validated before submitting new jobs.
93+
<img src="adm/rest_api.png" alt="slrum rest api" width="1000">
15794

158-
## Teardown Jupyter Notebook
95+
## Teardown
15996

160-
If you would like to tear down the notebook deployment, use the command below.
97+
To remove the Jupyter Notebook deployment when it is no longer needed, run the following command:
16198

16299
```bash
163100
./gcluster destroy af3-slurm-ipynb --auto-approve
@@ -167,3 +104,8 @@ If you would like to tear down the notebook deployment, use the command below.
167104
> If you do not destroy the Jupyter Notebook deployment, it may continue to incur costs.
168105
> Additionally, any Cloud Storage buckets you created (via the CLI or console) will not be automatically deleted. You are responsible for cleaning them up manually to avoid unnecessary charges.
169106
> For deleting the buckets consult [Delete buckets](https://cloud.google.com/storage/docs/deleting-buckets).
107+
108+
## Notes and Customization
109+
You can adjust the notebook setup behavior using blueprint variables in the deployment YAML.
110+
All configurations should be validated before running jobs.
111+
If further modifications to SLURM REST/API Server behavior are required, you must destroy and redeploy the cluster with the updated settings.

0 commit comments

Comments
 (0)