Skip to content

Commit c5c9299

Browse files
cboneticboneti
authored
Merge pull request GoogleCloudPlatform#4798 from gqiu-sycomp-com/feature/support-sycomp-storage
Support sycomp storage
2 parents 8c7c3a9 + a000341 commit c5c9299

7 files changed

Lines changed: 434 additions & 0 deletions

File tree

‎community/examples/sycomp/README.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# Sycomp Intelligent Data Storage Platform Blueprint
2+
3+
This document provides instructions on how to use a blueprint to deploy or expand a `Sycomp Intelligent Data Storage Platform` cluster and enable a `SLURM` cluster to access the data via `NFS` on Google Cloud Platform (GCP)
4+
using the Google Cluster Toolkit.
5+
6+
The directory contains four example blueprints that can be used to deploy or expand a `Sycomp Storage` cluster:
7+
8+
- sycomp-storage.yaml
9+
10+
Blueprint used to deploy a `Sycomp Storage` cluster consisting of 3 storage servers.
11+
12+
- sycomp-storage-expansion.yaml
13+
14+
Blueprint used to expand the `Sycomp Storage` cluster to 4 storage servers.
15+
16+
- sycomp-storage-ece.yaml
17+
18+
Blueprint used to deploy a `Sycomp Storage` cluster consisting of 7 storage servers using `ECE` (Erasure Code Edition) software RAID.
19+
20+
- sycomp-storage-slurm.yaml
21+
22+
Blueprint used to deploy a `SLURM` cluster and a `Sycomp Storage` cluster with three servers. The `SLURM` compute nodes are configured as `NFS` clients and have the ability to use the Sycomp Storage filesystem.
23+
24+
## Prerequisites
25+
26+
1. Google Cloud SDK is installed and configured.
27+
2. The Google Cloud Cluster Toolkit is set up [Set up Cluster Toolkit](https://cloud.google.com/cluster-toolkit/docs/setup/configure-environment).
28+
3. You have a Google Cloud project and the required permissions to create VPCs, and Compute Engine instances.
29+
4. The following APIs must be enabled in your project:
30+
- Compute Engine API (`compute.googleapis.com`)
31+
5. You have an SSH key pair (`~/.ssh/id_rsa` and `~/.ssh/id_rsa.pub` or similar). If you don't have one, you can generate it using `ssh-keygen -t rsa`.
32+
6. You have a valid **Customer Token** provided by Sycomp.
33+
7. You have a valid **Project access token** provided by Sycomp.
34+
35+
> **_NOTE:_** To get a customer and project access token, contact Sycomp (sycompstorage@sycomp.com).
36+
>
37+
> **_NOTE:_** To avoid repeatedly entering passwords, you can set `credential.helper` in git.
38+
39+
```shell
40+
git config credential.helper cache
41+
```
42+
43+
## Configuration
44+
45+
Before deploying you need to edit the blueprint yaml files and replace the placeholder values.
46+
47+
Required parameter updates for each blueprint:
48+
49+
- **`vars` block:**
50+
- `project_id`: Your Google Cloud project ID.
51+
- `deployment_name`: A unique name for this deployment (e.g., `sycomp-storage`).
52+
- `region`: The region where you want to deploy the cluster.
53+
- `zone`: The zone where you want to deploy the cluster.
54+
55+
- **`network1` module settings:**
56+
- `source`: `modules/network/vpc` is the default and deploys the cluster in a newly created network. To use an existing network, change `source` to `modules/network/pre-existing-vpc`.
57+
- `network_name`: A name for the new VPC network (e.g., `sycomp-net`).
58+
- `subnetwork_name`: A name for the new subnetwork (e.g., `sycomp-subnet`).
59+
- `allowed_ssh_ip_ranges`: A list of IP address ranges in CIDR format that
60+
are allowed to connect via SSH. **You must include the IP address of the
61+
machine you are running the deployment from.** For example: `["1.2.3.0/24"]`.
62+
63+
- **`sycomp-scale-gcp` module settings:**
64+
- `security.ssh.ssh_user_name`: The username for SSH access to the management node (e.g., `sycomp`).
65+
- `security.ssh.private_key`: The file path to your SSH private key (e.g., `~/.ssh/id_rsa`).
66+
- `security.ssh.public_key`: The file path to your SSH public key (e.g., `~/.ssh/id_rsa.pub`).
67+
- `security.customer_token.token`: Your Sycomp Customer Token. To get a customer token, contact Sycomp (`sycompstorage@sycomp.com`).
68+
- `name_prefix` (Optional): By default, `sycomp-scale-gcp` sets the value of `name_prefix` based on the `deployment_name`. Since `name_prefix` only accepts **alphanumeric** characters (letters and numbers), `sycomp-scale-gcp` automatically **removes any non-alphanumeric characters** (such as hyphens and underscores) before assigning the value. For example, a `deployment_name` of `sycomp-storage-1` would result in a `name_prefix` of `sycompstorage1`.
69+
- **`scale_config`** (Optional):
70+
- `scale_node_count`: The number of nodes in the Sycomp Storage cluster. Default is `3`.
71+
- `scale_volumes`: Configuration for the data disks. Default is 4 disks of 250GiB each per storage node.
72+
73+
- **`sycomp-scale-expansion` module settings:**
74+
- `name_prefix`: Must match the `name_prefix` of the cluster to be expanded. Note: if not set explicitly, `name_prefix` is derived from `deployment_name` by removing non-alphanumeric characters. You can find the correct value for an existing cluster by checking its Terraform outputs or by inspecting resource names in the Google Cloud Console.
75+
- `add_scale_nodes`: The number of nodes to be added.
76+
77+
## Deployment
78+
79+
Once the blueprint file (e.g., `sycomp-storage.yaml`) is configured, you can deploy the cluster from the blueprint by following these steps from your terminal.
80+
81+
1. **Authenticate with Google Cloud:**
82+
83+
```bash
84+
gcloud auth login
85+
gcloud auth application-default login
86+
```
87+
88+
2. **Create the deployment directory:**
89+
90+
```bash
91+
# Replace <blueprint-filename> with the specific blueprint filename (e.g., sycomp-storage.yaml).
92+
gcluster create community/examples/sycomp/<blueprint-filename>
93+
```
94+
95+
This command will create a new directory named after your `deployment_name` (e.g., `sycomp-storage`).
96+
97+
3. **Deploy the resources:**
98+
99+
```bash
100+
# The terraform init command will prompt for a username and password.
101+
# Enter any username and use the project access token from Sycomp as the password.
102+
# To get a project access token, contact Sycomp (sycompstorage@sycomp.com).
103+
#
104+
# Replace <deployment_name> with the `deployment_name` from your blueprint (e.g., "sycomp-storage").
105+
terraform -chdir=<deployment_name>/primary init
106+
gcluster deploy <deployment_name>
107+
```
108+
109+
This process will take several minutes as it provisions the Sycomp Storage cluster nodes.
110+
111+
## Cleanup
112+
113+
To remove all resources created by this blueprint, run the following command:
114+
115+
```bash
116+
gcluster destroy <deployment_name>
117+
```

‎community/examples/sycomp/sycomp-storage-ece.yaml‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Copyright 2024 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
---
16+
17+
blueprint_name: sycomp-storage-ece
18+
19+
vars:
20+
project_id: "" ## Set GCP Project ID Here
21+
deployment_name: "sycomp-ece" ## Set customized deployment name
22+
region: "" ## Set region
23+
zone: "" ## Set zone
24+
25+
# Documentation for each of the modules used below can be found at
26+
# https://github.com/GoogleCloudPlatform/hpc-toolkit/blob/main/modules/README.md
27+
28+
deployment_groups:
29+
- group: primary
30+
modules:
31+
32+
## Create network for sycomp storage
33+
- id: network1
34+
source: modules/network/vpc
35+
settings:
36+
network_name: "" ## Set network name
37+
subnetwork_name: "" ## Set subnetwork name
38+
# Source ip ranges for ssh, for example: [21.32.32.0/24].
39+
# In order to be able to login to the login node via ssh, the ip address
40+
# of this machine needs to be placed in the allowed_ssh_ip_ranges.
41+
allowed_ssh_ip_ranges: []
42+
43+
- id: sycomp-scale-gcp
44+
source: "git::https://gitlab.com/sycomp/cluster-toolkit.git//sycomp-scale?ref=main"
45+
use: [network1]
46+
settings:
47+
security:
48+
ssh:
49+
ssh_user_name: "" ## Set user name to ssh mgmt node, for example: "sycompacct".
50+
private_key: "" ## Set private key file, for example: ~/.ssh/id_rsa.
51+
public_key: "" ## Set public key file, for example: ~/.ssh/id_rsa.pub.
52+
customer_token:
53+
token: "" ## Set Your Customer Token.
54+
ece:
55+
use_ece: true ## Set deploy mode to ECE.
56+
ece_node_per_rg: 7 ## The number of nodes for each Recovery Group, When using ece_raid_code "4+2p" the recommended number of nodes per recovery group is 7.
57+
ece_raid_code: "" ## The raid code, for example: "4+2p".
58+
ece_block_size: "" ## Set block size, for example: "1m".
59+
60+
scale_config:
61+
scale_node_count: 7 ## Number of storage nodes to deploy, its value must be equal to or a multiple of ece_node_per_rg.
62+
scale_machine_type: "c3-standard-44" ## Instance type of the storage nodes, "c3-standard-44" is recommended value.
63+
scale_volumes:
64+
data-storage:
65+
count: 3 ## Number of disks to configure on each storage node. This value is configurable.
66+
size_in_gb: 250 ## Size of each disk in GB. Change '250' to your desired size.
67+
type: "hyperdisk-balanced" ## Type of persistent disk to use. This value is configurable.
68+
provisioned_iops: 30000 ## Provisioned IOPS per disk, required for Hyperdisk.
69+
provisioned_throughput: 500 ## Provisioned Throughput in MiB/sec per disk, required for Hyperdisk.

‎community/examples/sycomp/sycomp-storage-expansion.yaml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Copyright 2024 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
---
16+
17+
blueprint_name: sycomp-storage-expansion
18+
19+
vars:
20+
# Note: project_id, region, and zone must match the existing cluster.
21+
project_id: "" ## Set GCP Project ID of existing cluster
22+
deployment_name: "sycomp-expansion" ## Set a name for this expansion deployment
23+
region: "" ## Set region of existing cluster
24+
zone: "" ## Set zone of existing cluster
25+
26+
# Documentation for each of the modules used below can be found at
27+
# https://github.com/GoogleCloudPlatform/hpc-toolkit/blob/main/modules/README.md
28+
29+
deployment_groups:
30+
- group: primary
31+
modules:
32+
33+
- id: sycomp-scale-expansion
34+
source: "git::https://gitlab.com/sycomp/cluster-toolkit.git//sycomp-scale-expansion?ref=main"
35+
settings:
36+
name_prefix: "<name_prefix_of_existing_cluster>" ## Set to the name_prefix of the cluster to expand. See the README for details on how this value is determined.
37+
add_scale_nodes: 1 ## Number of storage nodes to add, This value is configurable.

‎community/examples/sycomp/sycomp-storage-slurm.yaml‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Copyright 2024 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
---
16+
17+
blueprint_name: sycomp-storage-slurm
18+
19+
vars:
20+
project_id: "" ## Set GCP Project ID Here
21+
deployment_name: "sycomp-slurm" ## Set a customized deployment name
22+
region: "" ## Set region
23+
zone: "" ## Set zone
24+
nfs_dir: "/gpfs/fs1/export1" ## Set the directory to export
25+
26+
# Documentation for each of the modules used below can be found at
27+
# https://github.com/GoogleCloudPlatform/hpc-toolkit/blob/main/modules/README.md
28+
29+
deployment_groups:
30+
- group: primary
31+
modules:
32+
33+
## Create network shared between sycomp storage cluster slurm cluster
34+
- id: network1
35+
source: modules/network/vpc
36+
settings:
37+
network_name: "" ## Set network name
38+
subnetwork_name: "" ## Set subnetwork name
39+
# Source ip ranges for ssh, for example: [21.32.32.0/24].
40+
# In order to be able to login to the login node via ssh, the ip address
41+
# of this machine needs to be placed in the allowed_ssh_ip_ranges.
42+
allowed_ssh_ip_ranges: []
43+
44+
- id: sycomp-scale-gcp
45+
source: "git::https://gitlab.com/sycomp/cluster-toolkit.git//sycomp-scale?ref=main"
46+
use: [network1]
47+
settings:
48+
security:
49+
ssh:
50+
ssh_user_name: "" ## Set user name to ssh mgmt node, for example: "sycompacct"
51+
private_key: "" ## Set private key file, for example: ~/.ssh/id_rsa
52+
public_key: "" ## Set public key file, for example: ~/.ssh/id_rsa.pub
53+
customer_token:
54+
token: "" ## Set Your Customer Token
55+
scale_config:
56+
scale_node_count: 3 ## Number of storage nodes to deploy. This value is configurable.
57+
scale_machine_type: "c3-standard-44" ## Instance type of the storage nodes. This value is configurable.
58+
scale_volumes:
59+
data-storage:
60+
count: 4 ## Number of disks to configure on each storage node. This value is configurable.
61+
size_in_gb: 250 ## Size of each disk in GB. Change '250' to your desired size.
62+
type: "hyperdisk-balanced" ## Type of persistent disk to use. This value is configurable.
63+
provisioned_iops: 30000 ## Provisioned IOPS per disk, required for Hyperdisk.
64+
provisioned_throughput: 500 ## Provisioned Throughput in MiB/sec per disk, required for Hyperdisk.
65+
66+
ces_config:
67+
enabled: true ## A value of true enables NFS service
68+
create_dns: true ## Create a round robin DNS entry that include all of the NFS Servers, one NFS server is created on each storage node.
69+
nfs_exports:
70+
- path: $(vars.nfs_dir)
71+
clients:
72+
- no_root_squash: true ## Disable root squashing. Set to false to enable(secure default).
73+
74+
## Set up slurm cluster and configure NFS client
75+
- id: compute_nodeset
76+
source: community/modules/compute/schedmd-slurm-gcp-v6-nodeset
77+
use: [network1]
78+
settings:
79+
node_count_dynamic_max: 4
80+
machine_type: n2-standard-2
81+
allow_automatic_updates: false
82+
83+
- id: compute_partition
84+
source: community/modules/compute/schedmd-slurm-gcp-v6-partition
85+
use:
86+
- compute_nodeset
87+
settings:
88+
is_default: true
89+
partition_name: compute
90+
91+
- id: slurm_login
92+
source: community/modules/scheduler/schedmd-slurm-gcp-v6-login
93+
use: [network1]
94+
settings:
95+
machine_type: n2-standard-4
96+
enable_login_public_ips: true
97+
98+
- id: slurm_controller
99+
source: community/modules/scheduler/schedmd-slurm-gcp-v6-controller
100+
use:
101+
- network1
102+
- compute_partition
103+
- slurm_login
104+
- sycomp-scale-gcp
105+
settings:
106+
enable_controller_public_ips: true

‎community/examples/sycomp/sycomp-storage.yaml‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Copyright 2024 Google LLC
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
---
16+
17+
blueprint_name: sycomp-storage
18+
19+
vars:
20+
project_id: "" ## Set GCP Project ID Here
21+
deployment_name: "sycomp-storage" ## Set customized deployment name
22+
region: "" ## Set region
23+
zone: "" ## Set zone
24+
25+
# Documentation for each of the modules used below can be found at
26+
# https://github.com/GoogleCloudPlatform/hpc-toolkit/blob/main/modules/README.md
27+
28+
deployment_groups:
29+
- group: primary
30+
modules:
31+
32+
## Create network shared between sycomp storage cluster
33+
- id: network1
34+
source: modules/network/vpc
35+
settings:
36+
network_name: "" ## Set network name
37+
subnetwork_name: "" ## Set subnetwork name
38+
# Source ip ranges for ssh, for example: [21.32.32.0/24].
39+
# In order to be able to login to the login node via ssh, the ip address
40+
# of this machine needs to be placed in the allowed_ssh_ip_ranges.
41+
allowed_ssh_ip_ranges: []
42+
43+
- id: sycomp-scale-gcp
44+
source: "git::https://gitlab.com/sycomp/cluster-toolkit.git//sycomp-scale?ref=main"
45+
use: [network1]
46+
settings:
47+
security:
48+
ssh:
49+
ssh_user_name: "" ## Set user name to ssh mgmt node, for example: "sycompacct"
50+
private_key: "" ## Set private key file, for example: ~/.ssh/id_rsa
51+
public_key: "" ## Set public key file, for example: ~/.ssh/id_rsa.pub
52+
customer_token:
53+
token: "" ## Set Your Customer Token
54+
scale_config:
55+
scale_node_count: 3 ## Number of storage nodes to deploy
56+
scale_machine_type: "c3-standard-44" ## Instance type of the storage nodes
57+
scale_volumes:
58+
data-storage:
59+
count: 4 ## Number of disks to configure on each storage node
60+
size_in_gb: 250 ## Size of each disk
61+
type: "hyperdisk-balanced" ## Type of persistent disk to use
62+
provisioned_iops: 30000 ## Provisioned IOPS per disk, required for Hyperdisk
63+
provisioned_throughput: 500 ## Provisioned Throughput in MiB/sec per disk, required for Hyperdisk
64+
afm_config:
65+
afm_node_count: 1 ## Number of AFM nodes to deploy
66+
afm_machine_type: "c3-standard-44" ## Instance type of the AFM nodes

0 commit comments

Comments
 (0)