Skip to content

Commit 0c22370

Browse files
cboneticboneti
authored
Merge pull request GoogleCloudPlatform#3509 from wiktorn/weka-client
Add module providing mount scripts for WEKA filesystems
2 parents 943eec5 + f78192e commit 0c22370

10 files changed

Lines changed: 620 additions & 39 deletions

File tree

‎community/modules/file-system/weka-client/README.md‎

Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
## Description
2+
3+
This module provides scripts for client installation and mounting [WEKA]
4+
filesystems. Client supports both UDP and DPDK modes and allows customization of
5+
mount parameters using Compute VM instance metadata.
6+
7+
For deploying Weka cluster please consult [WEKA installation on GCP].
8+
9+
[WEKA]: https://www.weka.io/
10+
[WEKA installation on GCP]: https://docs.weka.io/planning-and-installation/weka-installation-on-gcp
11+
12+
## Prerequisites
13+
14+
* up and running Weka cluster
15+
* running on a [supported OS](https://docs.weka.io/planning-and-installation/prerequisites-and-compatibility#operating-system)
16+
* [open firewall](https://docs.weka.io/planning-and-installation/prerequisites-and-compatibility#required-ports)
17+
between WEKA backend servers and clients
18+
* VPC peering configuration:
19+
* if clients share VPCs created for WEKA cluster, no additional configuration
20+
is necessary
21+
* if dedicated VPCs are in use for clients, then WEKA VPCs needs to be peered
22+
with VPCs that are used as:
23+
* primary interface on client
24+
* interfaces dedicated for DPDK client
25+
* if dedicated VPCs are in use for clients, then those VPCs needs to be peered
26+
with each other
27+
28+
## Mounting
29+
This example creates mount scripts that will mount `default` filesystem from
30+
`10.0.0.3` WEKA backend:
31+
32+
```yaml
33+
- id: wekafs
34+
source: community/modules/file-system/weka-client
35+
settings:
36+
local_mount: /scratch
37+
server_ip: 10.0.0.3
38+
remote_mount: default
39+
40+
- id: mount-at-startup
41+
source: modules/scripts/startup-script
42+
settings:
43+
runners: $(wekafs.runners)
44+
```
45+
46+
If you need to add mount script along other runners, remember to add all 4
47+
runners provided by this script as shown in this example:
48+
49+
```yaml
50+
- id: mount-at-startup
51+
source: modules/scripts/startup-script
52+
settings:
53+
runners:
54+
- $(wekafs.client_install_runner)
55+
- $(wekafs.mount_runner)
56+
- type: shell
57+
content: |
58+
#!/bin/bash
59+
60+
echo Sample
61+
destination: sample-script.sh
62+
```
63+
64+
To use the client within Slurm partition, with DPDK, remember to set additional
65+
networks, and configure metadata. In this example, all four additional interfaces
66+
are dedicated to WEKA DPDK
67+
68+
```yaml
69+
- id: c2_60_nodeset
70+
source: community/modules/compute/schedmd-slurm-gcp-v6-nodeset
71+
use:
72+
- network
73+
- mount-at-startup # as defined in previous examples
74+
settings:
75+
bandwidth_tier: virtio_enabled # Weka requires VirtIO, from WEKA 4.4.1, DPDK is also supported on gVNIC
76+
additional_networks:
77+
- subnetwork: weka-client-1
78+
nic_type: VIRTIO_NET
79+
- subnetwork: weka-client-2
80+
nic_type: VIRTIO_NET
81+
- subnetwork: weka-client-3
82+
nic_type: VIRTIO_NET
83+
- subnetwork: weka-client-4
84+
nic_type: VIRTIO_NET
85+
machine_type: c2-standard-60
86+
metadata:
87+
weka-data_interfaces: 1,2,3,4 # allocate interfaces 1, 2, 3 and 4 to DPDK
88+
weka-mode: dpdk
89+
weka-options: num_cores=4,dpdk_base_memory_mb=16
90+
node_conf:
91+
# From https://docs.weka.io/planning-and-installation/bare-metal/planning-a-weka-system-installation
92+
# do not set RealMem as this is set automatically by Cluster Toolkit
93+
CoreSpecCount: 4
94+
MemSpecLimit: 5120
95+
```
96+
97+
Due to the fact, that client installation takes ~6-7 minutes, if you use WEKA together with Slurm and do not bundle
98+
client in the instance image, you may need to increase the timeout for startups scripts.
99+
100+
```yaml
101+
- id: slurm_controller
102+
source: community/modules/scheduler/schedmd-slurm-gcp-v6-controller
103+
settings:
104+
compute_startup_scripts_timeout: 600
105+
login_startup_scripts_timeout: 600
106+
...
107+
- id: compute_partition
108+
source: community/modules/compute/schedmd-slurm-gcp-v6-partition
109+
settings:
110+
resume_timeout: 600
111+
...
112+
```
113+
114+
## Supported VM metadata options
115+
Client scripts do support following metadata keys:
116+
* `weka-mode` - one of `udp` or `dpdk`. Defaults to `udp`. Sets client mode.
117+
* `weka-data_interfaces` - comma separated list of interface identifiers,
118+
specifying which interfaces are dedicated for data plane. Set to `1` to
119+
dedicate second interface of instance for WEKA DPDK. Set to `2,5` to dedicate
120+
third and sixth interface of instance for WEKA DPDK.
121+
* `weka-mgmt_interface` - identifier of management interface, defaults to `0`,
122+
which means to use primary interface as management interface.
123+
* `weka-options` - additional [mount command options](https://docs.weka.io/weka-filesystems-and-object-stores/mounting-filesystems#mount-command-options)
124+
to pass to `mount` command
125+
126+
## Adding client to the OS image
127+
To save time during the mount command install and precompile DPDK drivers in the
128+
OS image. Following scripts compiles DPDK driver for currently running kernel.
129+
130+
```shell
131+
#!/bin/bash
132+
133+
set -e -o pipefail
134+
135+
echo Downloading and installing Weka client
136+
curl --max-time 10 "{{ weka backend endpoint }}/dist/v1/install" | sh
137+
WEKA_VERSION=$(weka -v | sed -e 's/^[^0-9]*//')
138+
echo Installing Weka version: ${WEKA_VERSION}
139+
weka version get "${WEKA_VERSION}"
140+
weka version set "${WEKA_VERSION}"
141+
# run setup for the second time, if it fails for the first time
142+
weka local setup weka || weka local setup weka
143+
weka version prepare "${WEKA_VERSION}"
144+
weka local stop
145+
weka local rm -f --all
146+
```
147+
148+
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
149+
## Requirements
150+
151+
| Name | Version |
152+
|------|---------|
153+
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.14.0 |
154+
155+
## Providers
156+
157+
No providers.
158+
159+
## Modules
160+
161+
No modules.
162+
163+
## Resources
164+
165+
No resources.
166+
167+
## Inputs
168+
169+
| Name | Description | Type | Default | Required |
170+
|------|-------------|------|---------|:--------:|
171+
| <a name="input_local_mount"></a> [local\_mount](#input\_local\_mount) | The mount point where the contents of the device may be accessed after mounting. | `string` | `"/mnt"` | no |
172+
| <a name="input_mount_options"></a> [mount\_options](#input\_mount\_options) | Mount options for filesystem shared by all clients. | `string` | `""` | no |
173+
| <a name="input_remote_mount"></a> [remote\_mount](#input\_remote\_mount) | Weka filesystem name. | `string` | n/a | yes |
174+
| <a name="input_server_ip"></a> [server\_ip](#input\_server\_ip) | Weka backend IP address used for bootstrapping. | `string` | `""` | no |
175+
176+
## Outputs
177+
178+
| Name | Description |
179+
|------|-------------|
180+
| <a name="output_client_install_runner"></a> [client\_install\_runner](#output\_client\_install\_runner) | Ansible runner that performs client installation needed to use file system. |
181+
| <a name="output_mount_runner"></a> [mount\_runner](#output\_mount\_runner) | Ansible runner that mounts the file system. |
182+
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

‎community/modules/file-system/weka-client/metadata.yaml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# Copyright 2025 "Google LLC"
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
---
15+
16+
spec:
17+
requirements:
18+
services: []

‎community/modules/file-system/weka-client/outputs.tf‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
/**
2+
* Copyright 2025 Google LLC
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
locals {
18+
template_args = {
19+
local_mount = var.local_mount
20+
mount_options = var.mount_options == "" ? "" : "-o ${var.mount_options}"
21+
remote_mount = var.remote_mount
22+
server_ip = var.server_ip
23+
service_name = "weka-mount${replace(var.local_mount, "/", "-")}"
24+
}
25+
mount_script = templatefile("${path.module}/templates/mount-weka.sh.tftpl", local.template_args)
26+
27+
mount_runner_ansible = {
28+
type = "ansible-local"
29+
content = templatefile(
30+
"${path.module}/templates/mount-weka.yaml.tftpl",
31+
merge(
32+
local.template_args,
33+
{ mount_weka_script = local.mount_script }
34+
)
35+
)
36+
destination = "mount_filesystem${replace(var.local_mount, "/", "_")}.yaml"
37+
}
38+
39+
client_install_runner = {
40+
type = "ansible-local"
41+
content = templatefile("${path.module}/templates/install-weka-client.yaml.tftpl", local.template_args)
42+
destination = "install_filesystem${replace(var.local_mount, "/", "_")}.yaml"
43+
}
44+
}
45+
46+
# currently WEKA mounts are not compatible with network_storage logic, as WEKA volumes needs to be mounted by
47+
# systemd script and not /etc/fstab entry, as the mount command needs to have network configuration which may change
48+
# between restarts
49+
#
50+
#output "network_storage" {
51+
# description = "Describes a remote network storage to be mounted by fs-tab."
52+
# value = {
53+
# server_ip = var.server_ip
54+
# remote_mount = var.remote_mount
55+
# local_mount = var.local_mount
56+
# fs_type = var.fs_type
57+
# mount_options = var.mount_options
58+
# client_install_runner = local.client_install_runner
59+
# mount_runner = local.mount_runner
60+
# }
61+
#}
62+
#
63+
output "client_install_runner" {
64+
description = "Ansible runner that performs client installation needed to use file system."
65+
value = local.client_install_runner
66+
}
67+
68+
output "mount_runner" {
69+
description = "Ansible runner that mounts the file system."
70+
value = local.mount_runner_ansible
71+
}

0 commit comments

Comments
 (0)