From 0c135ed8aef8248ef4d6a761377d749a743a9a99 Mon Sep 17 00:00:00 2001 From: Matt Richerson Date: Wed, 10 Sep 2025 11:44:10 -0500 Subject: [PATCH 1/2] Add documentation for RAID devices Signed-off-by: Matt Richerson --- docs/guides/index.md | 1 + docs/guides/redundant-allocations/readme.md | 38 +++++++++++ docs/guides/storage-profiles/readme.md | 72 +++++++++++++++++++++ mkdocs.yml | 1 + 4 files changed, 112 insertions(+) create mode 100644 docs/guides/redundant-allocations/readme.md diff --git a/docs/guides/index.md b/docs/guides/index.md index f6861581..9904a4a8 100644 --- a/docs/guides/index.md +++ b/docs/guides/index.md @@ -18,6 +18,7 @@ * [Directive Breakdown](directive-breakdown/readme.md) * [User Interactions](user-interactions/readme.md) * [System Storage](system-storage/readme.md) +* [Redundant Allocations](redundant-allocations/readme.md) ## NNF User Containers diff --git a/docs/guides/redundant-allocations/readme.md b/docs/guides/redundant-allocations/readme.md new file mode 100644 index 00000000..630046fa --- /dev/null +++ b/docs/guides/redundant-allocations/readme.md @@ -0,0 +1,38 @@ +--- +authors: Matt Richerson +categories: provisioning +--- + +# Redundant Allocations + +## Background + +Lustre, XFS, and Raw allocations can be created using a redundant configuration either through zpools or LVM. This guide documents how these configurations might be used. + +## Creating a Redundant Allocation + +An allocation may choose to use a redundant configuration to allow access to data even when a drive has failed. In addition, rebuild commands can be specified to allow a replacement drive to be added to the RAID. + +The [Storage Profiles](../storage-profiles/readme.md) section details the command lines needed to create an allocation using a RAID device. + +## Rebuilding a Redundant Allocation + +Choosing whether to allow a redundant allocation to be rebuilt with a new drive depends on the file system and allocation type. + +### LVM + +For XFS and Raw allocations using LVM, the RAID device can only be rebuilt when the Rabbit node has the LV activated. For `jobdw` allocations, this means that the RAID cannot be rebuilt while the compute nodes have the LV activated between `PreRun` and `PostRun`. For this reason, the most common way to use RAID configurations in this situation will be without rebuilding enabled. + +If an XFS or Raw allocation is made with `create_persistent`, then the rebuild commands should be specified to allow the RAID to be rebuilt when no compute nodes have the LV activated. + +### Zpool + +For Lustre allocations using zpool, there are no restrictions on when rebuild commands can be used. The RAID can be rebuilt for job and persistent instances regardless of whether the compute nodes have the file system mounted. + +### Replacing a Drive + +If a drive has failed and needs to be replaced, the Rabbit-p and Rabbit-s should be powered off to replace the drive. After powering the Rabbit back on, the `nnf-node-manager` pod will restart on the Rabbit. On initialization, Rabbit software will find the new drive and add an NVMe namespace for each allocation. If the rebuild commands have been specified, the Rabbit software will run them to add the new NVMe namespace into the RAID device. + +## Allocation Status + +If the allocation is a `PersistentStorageInstance`, the status of the RAID device is available in the `Status.State` field. If all the RAIDs in the persistent storage are healthy, `Status.State=Active`. If any of the RAIDs are degraded, `Status.State=Degraded`. \ No newline at end of file diff --git a/docs/guides/storage-profiles/readme.md b/docs/guides/storage-profiles/readme.md index 94871fce..a7f062fa 100644 --- a/docs/guides/storage-profiles/readme.md +++ b/docs/guides/storage-profiles/readme.md @@ -286,6 +286,78 @@ The `count` field may be useful when creating a persistent file system since the In general, `scale` gives a simple way for users to get a filesystem that has performance consistent with their job size. `count` is useful for times when a user wants full control of the file system layout. +### RAID Configurations + +Allocations can be set up to use a RAID device to provide redundancy in the event of a drive failure. Optionally, commands can be specified to rebuild the RAID device after a replacement drive has been added. The storage profile parameters differ depending on whether the allocation is using LVM or zpool. + +#### Zpool + +To create a Lustre confiuration with a redundant zpool, the `raidz` option should be added to the `zpoolCreate` command. To allow the zpool to be rebuilt with a new drive, the `zpoolReplace` command should be included. + +The example below shows a RAID configuration for the OST, however, the same options can be specified for any of the Lustre targets. + +```yaml +apiVersion: nnf.cray.hpe.com/v1alpha8 +kind: NnfStorageProfile +metadata: + name: lustre-raid-example + namespace: nnf-system +data: +[...] + lustreStorage: +[...] + ostCommandlines: + mkfs: --ost --backfstype=$BACKFS --fsname=$FS_NAME --mgsnode=$MGS_NID --index=$INDEX + --mkfsoptions="nnf:jobid=$JOBID" $ZVOL_NAME + mountTarget: $ZVOL_NAME $MOUNT_PATH + postActivate: + - mountpoint $MOUNT_PATH + zpoolCreate: -O canmount=off -o cachefile=none $POOL_NAME raidz $DEVICE_LIST + zpoolReplace: $POOL_NAME $OLD_DEVICE $NEW_DEVICE +``` + +#### LVM + +A RAID logical volume can be used with XFS and Raw allocations. gfs2 allocations can not use RAID logical volumes because the LV is shared. + +To create a redundant LV, `--type raid[x]` and `--nosync` should be specified in the `lvcreate` command. Also, the `--stripes` parameter should be adjusted accordingly to specify the number of data stripes. For `raid5`, `$DEVICE_NUM-1` is used. + +To allow the LV to be rebuilt after a replacement drive is added, `vgExtend`, `lvRepair`, and `vgReduce` should be specified in the `lvmRebuild` section. + +```yaml +apiVersion: nnf.cray.hpe.com/v1alpha8 +kind: NnfStorageProfile +metadata: + name: xfs-raid-example + namespace: nnf-system +data: +[...] + xfsStorage: + commandlines: + lvChange: + activate: --activate y $VG_NAME/$LV_NAME + deactivate: --activate n $VG_NAME/$LV_NAME + lvmRebuild: + vgExtend: $VG_NAME $DEVICE + vgReduce: --removemissing $VG_NAME + lvRepair: $VG_NAME/$LV_NAME + lvCreate: --activate n --zero n --nosync --type raid5 --extents $PERCENT_VG --stripes $DEVICE_NUM-1 + --stripesize=32KiB --name $LV_NAME $VG_NAME + lvRemove: $VG_NAME/$LV_NAME + mkfs: $DEVICE + mountCompute: $DEVICE $MOUNT_PATH + mountRabbit: $DEVICE $MOUNT_PATH + postMount: + - chown $USERID:$GROUPID $MOUNT_PATH + pvCreate: $DEVICE + pvRemove: $DEVICE + sharedVg: true + vgChange: + lockStart: --lock-start $VG_NAME + lockStop: --lock-stop $VG_NAME + vgCreate: --shared --addtag $JOBID $VG_NAME $DEVICE_LIST + vgRemove: $VG_NAME + ## Command Line Variables ### global diff --git a/mkdocs.yml b/mkdocs.yml index 32d83c66..a01d9cba 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -28,6 +28,7 @@ nav: - 'Switch a Node From Worker to Master': 'guides/node-management/worker-to-master.md' - 'Directive Breakdown': 'guides/directive-breakdown/readme.md' - 'System Storage': 'guides/system-storage/readme.md' + - 'Redundant Allocations': 'guides/redundant-allocations/readme.md' - 'Repo Guides': - 'Releasing NNF Software': 'repo-guides/release-nnf-sw/release-all.md' - 'CRD Version Bumper': 'repo-guides/crd-bumper/readme.md' From e3ab7ee8f6fb6c567be9e4ca281def2752b564ae Mon Sep 17 00:00:00 2001 From: Matt Richerson Date: Wed, 10 Sep 2025 13:55:23 -0500 Subject: [PATCH 2/2] review comments Signed-off-by: Matt Richerson --- docs/guides/redundant-allocations/readme.md | 4 ++-- docs/guides/storage-profiles/readme.md | 9 +++++---- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/guides/redundant-allocations/readme.md b/docs/guides/redundant-allocations/readme.md index 630046fa..71602856 100644 --- a/docs/guides/redundant-allocations/readme.md +++ b/docs/guides/redundant-allocations/readme.md @@ -27,11 +27,11 @@ If an XFS or Raw allocation is made with `create_persistent`, then the rebuild c ### Zpool -For Lustre allocations using zpool, there are no restrictions on when rebuild commands can be used. The RAID can be rebuilt for job and persistent instances regardless of whether the compute nodes have the file system mounted. +For Lustre allocations using zpool, there are no restrictions on when rebuild commands can be used. The RAID can be rebuilt for job and persistent instances regardless of whether the compute nodes have the file system mounted. This is possible because the Lustre targets are only mounted on the Rabbits. ### Replacing a Drive -If a drive has failed and needs to be replaced, the Rabbit-p and Rabbit-s should be powered off to replace the drive. After powering the Rabbit back on, the `nnf-node-manager` pod will restart on the Rabbit. On initialization, Rabbit software will find the new drive and add an NVMe namespace for each allocation. If the rebuild commands have been specified, the Rabbit software will run them to add the new NVMe namespace into the RAID device. +If a drive fails and must be replaced, the Rabbit-p and Rabbit-s should be powered off to replace the drive. After replacing the drive and powering the Rabbit back on, the `nnf-node-manager` pod restarts on the Rabbit. On initialization, Rabbit software locates the new drive and adds an NVMe namespace for each allocation. If rebuild commands have been specified, the Rabbit software will run them to add the new NVMe namespace into the RAID device and restore the correct data. ## Allocation Status diff --git a/docs/guides/storage-profiles/readme.md b/docs/guides/storage-profiles/readme.md index a7f062fa..7a4e52ea 100644 --- a/docs/guides/storage-profiles/readme.md +++ b/docs/guides/storage-profiles/readme.md @@ -288,11 +288,11 @@ In general, `scale` gives a simple way for users to get a filesystem that has pe ### RAID Configurations -Allocations can be set up to use a RAID device to provide redundancy in the event of a drive failure. Optionally, commands can be specified to rebuild the RAID device after a replacement drive has been added. The storage profile parameters differ depending on whether the allocation is using LVM or zpool. +Allocations can be set up to use a RAID device to provide continued access in the event of a drive failure. Optionally, commands can be specified to rebuild the RAID device after a replacement drive has been added. The storage profile parameters differ depending on whether the allocation is using LVM or zpool. #### Zpool -To create a Lustre confiuration with a redundant zpool, the `raidz` option should be added to the `zpoolCreate` command. To allow the zpool to be rebuilt with a new drive, the `zpoolReplace` command should be included. +To create a Lustre confiuration with a redundant zpool, the `raidz` option is required in `zpoolCreate` command. To allow the zpool to be rebuilt with a new drive, the `zpoolReplace` command is required. The example below shows a RAID configuration for the OST, however, the same options can be specified for any of the Lustre targets. @@ -318,11 +318,12 @@ data: #### LVM -A RAID logical volume can be used with XFS and Raw allocations. gfs2 allocations can not use RAID logical volumes because the LV is shared. +A RAID logical volume can be used with XFS and Raw allocations. +NOTE: gfs2 allocations cannot use RAID logical volumes because the LV is shared. To create a redundant LV, `--type raid[x]` and `--nosync` should be specified in the `lvcreate` command. Also, the `--stripes` parameter should be adjusted accordingly to specify the number of data stripes. For `raid5`, `$DEVICE_NUM-1` is used. -To allow the LV to be rebuilt after a replacement drive is added, `vgExtend`, `lvRepair`, and `vgReduce` should be specified in the `lvmRebuild` section. +To allow the LV to rebuild after a drive is replaced, `vgExtend`, `lvRepair`, and `vgReduce` should be specified in the `lvmRebuild` section. ```yaml apiVersion: nnf.cray.hpe.com/v1alpha8