You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/guides/storage-profiles/readme.md
+78-29Lines changed: 78 additions & 29 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -48,7 +48,7 @@ kubectl get nnfstorageprofiles -n nnf-system
48
48
49
49
Example output:
50
50
51
-
```
51
+
```text
52
52
NAME DEFAULT AGE
53
53
default true 14d
54
54
high-perf false 7d
@@ -199,6 +199,9 @@ data:
199
199
userCommands:
200
200
postSetup:
201
201
- chown $USERID:$GROUPID $MOUNT_PATH
202
+
# Optional: run diagnostics after block device teardown (Rabbit only)
203
+
# postTeardown:
204
+
# - /usr/bin/dlm-diagnostics.sh $VG_NAME
202
205
203
206
capacityScalingFactor: "1.0"
204
207
allocationPadding: 300MiB
@@ -338,7 +341,7 @@ Target layout options control how Lustre targets (MGT, MDT, OST) are distributed
338
341
Each target type (mgtOptions, mdtOptions, mgtMdtOptions, ostOptions) supports:
339
342
340
343
| Option | Description |
341
-
|--------|-------------|
344
+
|-------- | -------------|
342
345
| `count` | Static number of targets to create |
343
346
| `scale` | Dynamic value (1-10) that the WLM uses to determine target count |
344
347
| `colocateComputes` | If true, targets are placed on Rabbits connected to job's compute nodes |
@@ -349,30 +352,35 @@ Each target type (mgtOptions, mdtOptions, mgtMdtOptions, ostOptions) supports:
349
352
### Understanding colocateComputes
350
353
351
354
When `colocateComputes: true`:
355
+
352
356
- Storage is restricted to Rabbit nodes with physical connections to the job's compute nodes
353
357
- This typically means Rabbits in the same chassis as the compute nodes
354
358
- Best for minimizing network hops and maximizing bandwidth
355
359
356
360
When `colocateComputes: false`:
361
+
357
362
- Storage can be placed on any available Rabbit node
358
363
- Useful for separating metadata targets from data targets
359
364
- Required for `create_persistent` directives since they may not have compute nodes
360
365
361
366
### Scale vs Count
362
367
363
368
**Scale** is useful when you want storage to automatically adjust based on job size:
369
+
364
370
- Value of 1: Minimum targets needed to satisfy capacity
365
371
- Value of 10: Maximum targets, potentially one per Rabbit connected to the job
366
372
- The WLM interprets scale values based on allocation size, compute count, and Rabbit count
367
373
368
374
**Count** is useful when you need precise control:
375
+
369
376
- Specific number of targets regardless of job size
370
377
- Consistent performance characteristics across different jobs
371
378
- Useful for single-shared-file workloads with low metadata requirements
372
379
373
380
### Example Layouts
374
381
375
382
**High-performance scaled to job size:**
383
+
376
384
```yaml
377
385
ostOptions:
378
386
scale: 10
@@ -383,6 +391,7 @@ mdtOptions:
383
391
```
384
392
385
393
**Static Configuration:**
394
+
386
395
```yaml
387
396
ostOptions:
388
397
count: 4
@@ -461,7 +470,7 @@ There are two categories of user commands with different execution contexts:
461
470
The user commands in `blockDeviceCommands` and `fileSystemCommands` are run when storage is being activated/deactivated and mounted/unmounted for use:
462
471
463
472
| Location | Workflow Phase | Description |
464
-
|----------|----------------|-------------|
473
+
|---------- | ---------------- | -------------|
465
474
| Rabbit | DataIn | Storage is activated and mounted for data staging into the allocation |
466
475
| Rabbit | DataOut | Storage is activated and mounted for data staging out of the allocation |
467
476
| Rabbit | PreRun | Storage is activated and mounted for access in a user container |
@@ -470,6 +479,7 @@ The user commands in `blockDeviceCommands` and `fileSystemCommands` are run when
470
479
| Compute | PostRun | Storage is unmounted and deactivated after the user's application completes |
471
480
472
481
These commands are useful for operations that need to happen each time storage is accessed, such as:
482
+
473
483
- Setting up environment-specific configurations
474
484
- Running health checks before/after use
475
485
- Synchronizing data or caches
@@ -537,13 +547,15 @@ xfsStorage:
537
547
The `userCommands` section at the storage level (e.g., `xfsStorage.userCommands`) contains commands that run during the **setup and teardown phases** of the workflow. These run on the Rabbit nodes when storage is first provisioned and when it is finally destroyed.
538
548
539
549
| Command | Phase | Description |
540
-
|---------|-------|-------------|
550
+
|--------- | ------- | -------------|
541
551
| `postSetup` | Setup | Runs after storage is fully provisioned and the file system is mounted on the Rabbit |
542
552
| `preTeardown` | Teardown | Runs before storage is destroyed, while the file system is still mounted |
553
+
| `postTeardown` | Teardown | Runs after storage is destroyed — file system and block devices are already removed. Runs on the Rabbit only |
543
554
| `postActivate` | Setup | Runs after the file system is activated during initial setup |
544
555
| `preDeactivate` | Teardown | Runs before the file system is deactivated during final teardown |
545
556
546
557
These commands are useful for one-time operations such as:
558
+
547
559
- Setting ownership and permissions on newly created storage
548
560
- Initializing directory structures
549
561
- Cleaning up or archiving data before destruction
@@ -560,6 +572,11 @@ xfsStorage:
560
572
# Run once before storage is torn down (file system still mounted)
561
573
preTeardown:
562
574
- echo "Final cleanup of $MOUNT_PATH"
575
+
576
+
# Run once after storage is destroyed (file system and block devices already removed)
577
+
# Runs on the Rabbit only. $MOUNT_PATH is NOT available — use block device variables.
578
+
postTeardown:
579
+
- echo "Post-teardown check for $VG_NAME"
563
580
564
581
# Run once after file system is activated during setup
| `$FS_NAME` | Lustre fsname picked by NNF software |
724
743
| `$MGS_NID` | NID of the MGS |
725
744
| `$ZVOL_NAME` | ZFS volume name (`pool/dataset`) |
@@ -730,7 +749,7 @@ Available in all commands:
730
749
#### Lustre Client
731
750
732
751
| Variable | Description |
733
-
|----------|-------------|
752
+
|---------- | -------------|
734
753
| `$MGS_NID` | NID of the MGS |
735
754
| `$FS_NAME` | File system name |
736
755
| `$MOUNT_PATH` | Client mount path |
@@ -745,7 +764,7 @@ Available in all commands:
745
764
For system storage allocations:
746
765
747
766
| Variable | Description |
748
-
|----------|-------------|
767
+
|---------- | -------------|
749
768
| `$COMPUTE_HOSTNAME` | Hostname of the compute node using the allocation |
750
769
751
770
### User Command Variables
@@ -757,7 +776,7 @@ Different variables are available depending on which user command hook is being
757
776
The following variables are available to `blockDeviceCommands.rabbitCommands.userCommands` and `blockDeviceCommands.computeCommands.userCommands` (preActivate, postActivate, preDeactivate, postDeactivate):
758
777
759
778
| Variable | Description |
760
-
|----------|-------------|
779
+
|---------- | -------------|
761
780
| `$JOBID` | Job ID from the Workflow |
762
781
| `$USERID` | User ID of the job submitter |
763
782
| `$GROUPID` | Group ID of the job submitter |
@@ -769,7 +788,7 @@ The following variables are available to `blockDeviceCommands.rabbitCommands.use
769
788
The following variables are available to `fileSystemCommands.rabbitCommands.userCommands` and `fileSystemCommands.computeCommands.userCommands` (preMount, postMount, preUnmount, postUnmount):
770
789
771
790
| Variable | Description |
772
-
|----------|-------------|
791
+
|---------- | -------------|
773
792
| `$JOBID` | Job ID from the Workflow |
774
793
| `$USERID` | User ID of the job submitter |
775
794
| `$GROUPID` | Group ID of the job submitter |
@@ -781,18 +800,34 @@ The following variables are available to `fileSystemCommands.rabbitCommands.user
781
800
The following variables are available to `userCommands` at the storage level (postSetup, preTeardown, postActivate, preDeactivate):
782
801
783
802
| Variable | Description |
784
-
|----------|-------------|
803
+
|---------- | -------------|
785
804
| `$JOBID` | Job ID from the Workflow |
786
805
| `$USERID` | User ID of the job submitter |
787
806
| `$GROUPID` | Group ID of the job submitter |
788
807
| `$MOUNT_PATH` | Path where the file system is mounted |
789
808
809
+
For `postTeardown`, the file system and block devices have already been destroyed, so `$MOUNT_PATH` is **not** available. Instead, the following block device and workflow variables are available:
810
+
811
+
| Variable | Description |
812
+
| ---------- | ------------- |
813
+
| `$VG_NAME` | Volume group name |
814
+
| `$LV_NAME` | Logical volume name |
815
+
| `$LV_INDEX` | Allocation index |
816
+
| `$LV_SIZE` | Logical volume size |
817
+
| `$DEVICE_NUM` | Number of physical devices |
818
+
| `$DEVICE_LIST` | Space-separated list of device paths |
819
+
| `$JOBID` | Job ID from the Workflow |
820
+
| `$USERID` | User ID of the job submitter |
821
+
| `$GROUPID` | Group ID of the job submitter |
822
+
| `$ALLOCATION_COUNT` | Number of allocations on this Rabbit |
823
+
| `$HOST_COUNT` | Total host count (computes + Rabbit) |
824
+
790
825
#### Lustre Target User Commands
791
826
792
827
The following variables are available to Lustre target commands (mgtCommandlines, mdtCommandlines, mgtMdtCommandlines, ostCommandlines) for postActivate and preDeactivate:
793
828
794
829
| Variable | Description |
795
-
|----------|-------------|
830
+
|---------- | -------------|
796
831
| `$JOBID` | Job ID from the Workflow |
797
832
| `$MOUNT_PATH` | Path where the target is mounted |
798
833
| `$FS_NAME` | Lustre file system name |
@@ -805,7 +840,7 @@ The following variables are available to `clientCommandLines` user commands:
@@ -816,7 +851,7 @@ The following variables are available to `clientCommandLines` user commands:
816
851
**For rabbitPostSetup and rabbitPreTeardown:**
817
852
818
853
| Variable | Description |
819
-
|----------|-------------|
854
+
|---------- | -------------|
820
855
| `$JOBID` | Job ID from the Workflow |
821
856
| `$USERID` | User ID of the job submitter |
822
857
| `$GROUPID` | Group ID of the job submitter |
@@ -829,12 +864,27 @@ The following variables are available to `clientCommandLines` user commands:
829
864
| `$NUM_OSTS` | Number of OSTs |
830
865
| `$NUM_NNFNODES` | Number of NNF nodes |
831
866
867
+
**For rabbitPostTeardown:**
868
+
869
+
Runs on the Rabbit after the Lustre file system and zpool are destroyed. `$MOUNT_PATH` and Lustre-specific variables are **not** available. The following variables are available:
870
+
871
+
| Variable | Description |
872
+
| ---------- | ------------- |
873
+
| `$POOL_NAME` | Zpool name |
874
+
| `$DEVICE_NUM` | Number of physical devices |
875
+
| `$DEVICE_LIST` | Space-separated list of device paths |
876
+
| `$JOBID` | Job ID from the Workflow |
877
+
| `$USERID` | User ID of the job submitter |
878
+
| `$GROUPID` | Group ID of the job submitter |
879
+
| `$ALLOCATION_COUNT` | Number of allocations on this Rabbit |
880
+
| `$HOST_COUNT` | Total host count (computes + Rabbit) |
881
+
832
882
#### Lustre preMountMGTCommands
833
883
834
884
The following variables are available to `preMountMGTCommands`:
835
885
836
886
| Variable | Description |
837
-
|----------|-------------|
887
+
|---------- | -------------|
838
888
| `$JOBID` | Job ID from the Workflow |
839
889
| `$FS_NAME` | Lustre file system name |
840
890
@@ -844,8 +894,7 @@ The `variableOverride` fields in a storage profile allow overriding the default
844
894
845
895
When a variable is overridden, the NNF software uses the new value internally when manipulating block devices and file systems; not just for expanding `$VARIABLE` references in command lines. For example, overriding `$LV_NAME` changes the actual logical volume name that the NNF software uses to perform checks and run internal commands.
846
896
847
-
Variables are expanded recursively so that variable values can reference other variables. This allows building values based on job, workflow, and directive identifiers in a similar way to what the NNF software does internally.
848
-
897
+
Variables are expanded recursively so that variable values can reference other variables. This allows building values based on job, workflow, and directive identifiers in a similar way to what the NNF software does internally.
849
898
850
899
#### Overridable Variables by File System Type
851
900
@@ -861,7 +910,7 @@ xfsStorage:
861
910
```
862
911
863
912
| Variable | Default | Description |
864
-
|----------|---------|-------------|
913
+
|---------- | --------- | -------------|
865
914
| `$JOBID` | From workflow | Job ID from the workflow |
866
915
| `$USERID` | From workflow | User ID of the job submitter |
867
916
| `$GROUPID` | From workflow | Group ID of the job submitter |
0 commit comments