|
| 1 | +--- |
| 2 | +authors: Blake Devcich <blake.devcich@hpe.com> |
| 3 | +categories: provisioning |
| 4 | +--- |
| 5 | + |
| 6 | +# Global Lustre |
| 7 | + |
| 8 | +## Background |
| 9 | + |
| 10 | +Adding global lustre to rabbit systems allows access to external file systems. This is primarily |
| 11 | +used for Data Movement, where a user can perform `copy_in` and `copy_out` directives with global |
| 12 | +lustre being the source and destination, respectively. |
| 13 | + |
| 14 | +Global lustre fileystems are represented by the `lustrefilesystems` resource in Kubernetes: |
| 15 | + |
| 16 | +```shell |
| 17 | +$ kubectl get lustrefilesystems -A |
| 18 | +NAMESPACE NAME FSNAME MGSNIDS AGE |
| 19 | +default mylustre mylustre 10.1.1.113@tcp 20d |
| 20 | +``` |
| 21 | + |
| 22 | +An example resource is as follows: |
| 23 | + |
| 24 | +```yaml |
| 25 | +apiVersion: lus.cray.hpe.com/v1alpha1 |
| 26 | +kind: LustreFileSystem |
| 27 | +metadata: |
| 28 | + name: mylustre |
| 29 | + namespace: default |
| 30 | +spec: |
| 31 | + mgsNids: 10.1.1.100@tcp |
| 32 | + mountRoot: /p/mylustre |
| 33 | + name: mylustre |
| 34 | + namespaces: |
| 35 | + default: |
| 36 | + modes: |
| 37 | + - ReadWriteMany |
| 38 | +``` |
| 39 | +
|
| 40 | +## Namespaces |
| 41 | +
|
| 42 | +Note the `spec.namespaces` field. For each namespace listed, the `lustre-fs-operator` creates a |
| 43 | +PV/PVC pair in that namespace. This allows pods in that namespace to access global lustre. The |
| 44 | +`default` namespace should appear in this list. This makes the `lustrefilesystem` resource available |
| 45 | +to the `default` namespace, which makes it available to containers (e.g. container workflows) |
| 46 | +running in the `default` namespace. The `nnf-dm-system` namespace is added automatically - no need |
| 47 | +to specify that manually here. |
| 48 | + |
| 49 | +The `lustrefilesystem` resource itself should be created in the `default` namespace (i.e. |
| 50 | +`metadata.namespace`). |
| 51 | + |
| 52 | +## NNF Data Movement Manager |
| 53 | + |
| 54 | +The NNF Data Movement Manager is responsible for monitoring `lustrefilesystem` resources to mount |
| 55 | +(or umount) the global lustre filesystem in each of the NNF DM Worker pods. These pods run on each |
| 56 | +of the NNF nodes. This means with each addition or removal of `lustrefilesystems` resources, the DM |
| 57 | +worker pods restart to adjust their mount points. |
| 58 | + |
| 59 | +The NNF Data Movement Manager also places a finalizer on the `lustrefilesystem` resource to indicate |
| 60 | +that the resource is in use by Data Movement. This is to prevent the PV/PVC being deleted while they |
| 61 | +are being used by pods. |
| 62 | + |
| 63 | +## Adding Global Lustre |
| 64 | + |
| 65 | +As mentioned previously, the NNF Data Movement Manager monitors these resources and automatically |
| 66 | +adds the `nnf-dm-system` namespace to all `lustrefilesystem` resources. Once this happens, a PV/PVC |
| 67 | +is created for the `nnf-dm-system` namespace to access global lustre. The Manager updates the NNF DM |
| 68 | +Worker pods, which are then restarted to mount the global lustre file system. |
| 69 | + |
| 70 | +## Removing Global Lustre |
| 71 | + |
| 72 | +When a `lustrefilesystem` is deleted, the NNF DM Manager takes notice and starts to unmount the file |
| 73 | +system from the DM Worker pods - causing another restart of the DM Worker pods. Once this is |
| 74 | +finished, the DM finalizer is removed from the `lustrefilesystem` resource to signal that it is no |
| 75 | +longer in use by Data Movement. |
| 76 | + |
| 77 | +If a `lustrefilesystem` does not delete, check the finalizers to see what might still be using it. |
| 78 | +It is possible to get into a situation where `nnf-dm` has been undeployed, so there is nothing to |
| 79 | +remove the DM finalizer from the `lustrefilesystem` resource. If that is the case, then manually |
| 80 | +remove the DM finalizer so the deletion of the `lustrefilesystem` resource can continue. |
0 commit comments