Merge branch 'main' into release-v0.0.7

Author: ajfloeder

.github/workflows/publish-main.yaml

@@ -15,6 +15,7 @@ jobs:
         uses: actions/checkout@v3
         with:
           fetch-depth: 0
+          submodules: 'true'
 
       - name: Set up Python
         uses: actions/setup-python@v3

.github/workflows/release.yaml

@@ -15,6 +15,7 @@ jobs:
         uses: actions/checkout@v3
         with:
           fetch-depth: 0
+          submodules: 'true'
 
       - name: Set up Python
         uses: actions/setup-python@v3

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "external/nnf-dm"]
+	path = external/nnf-dm
+	url = git@github.com:NearNodeFlash/nnf-dm.git

README.md

@@ -1,5 +1,13 @@
 # NearNodeFlash Pages
 
+## Retrieve the submodules
+
+There are references to files in other repos, so update the submodules for them.
+
+```bash
+git submodule update external/nnf-dm
+```
+
 ## Create mkdocs Environment
 
 ```bash

docs/guides/compute-daemons/readme.md

@@ -30,7 +30,7 @@ NNF software defines a Kubernetes Service Account for granting communication pri
 
 | Compute Daemon | Service Account | Namespace |
 | -------------- | --------------- | --------- |
-| Client Mount   | dws-operator-controller-manager | dws-operator-system |
+| Client Mount   | dws-controller-manager | dws-system |
 | Data Movement  | nnf-dm-controller-manager | nnf-dm-system |
 
 ```bash

docs/guides/data-movement/copy-offload-api.html

@@ -0,0 +1 @@
+../../../external/nnf-dm/daemons/compute/copy-offload-api.html

docs/guides/data-movement/readme.md

@@ -62,23 +62,26 @@ mpirun --allow-run-as-root --hostfile $HOSTFILE dcp --progress 1 --uid $UID --gi
 
 ### Profiles
 
-**Note:** This feature is not fully implemented, but is present in the `nnf-dm-config` config map.
-Only the `default` profile is used, and it must be present in the configuration. Once the feature is
-implemented, a user will be able to select a profile using #DW directives and/or the Copy Offload
-API. Right now, users can add additional profiles into the config map, but the only way to use them
-would be to rename one of them to be `default`.
+Profiles can be specified in the in the `nnf-dm-config` config map. Users are able to select a
+profile using #DW directives (e.g .`copy_in profile=my-dm-profile`) and the Copy Offload API. If no
+profile is specified, the `default` profile is used. This default profile must exist in the config
+map.
 
 `slots`, `maxSlots`, and `command` can be stored in Data Movement profiles. These profiles are
-available for a quick way to switch between different settings for a particular workflow.
+available to quickly switch between different settings for a particular workflow.
 
-Example profile:
+Example profiles:
 
 ```yaml
 profiles:
   default:
       slots: 8
       maxSlots: 0
       command: mpirun --allow-run-as-root --hostfile $HOSTFILE dcp --progress 1 --uid $UID --gid $GID $SRC $DEST
+  no-xattrs:
+      slots: 8
+      maxSlots: 0
+      command: mpirun --allow-run-as-root --hostfile $HOSTFILE dcp --progress 1 --xattrs none --uid $UID --gid $GID $SRC $DEST
 ```
 
 ## Copy Offload API Daemon
@@ -87,5 +90,20 @@ The `CreateRequest` API call that is used to create Data Movement with the Copy
 options to allow a user to specify some options for that particular Data Movement. These settings
 are on a per-request basis.
 
-See the [CreateRequest API](https://github.com/NearNodeFlash/nnf-dm/blob/master/daemons/compute/Readme.md#create-request)
+See the [DataMovementCreateRequest API](copy-offload-api.html#datamovement.DataMovementCreateRequest)
 definition for what can be configured.
+
+## SELinux and Data Movement
+
+Careful consideration must be taken when enabling SELinux on compute nodes. Doing so will result in
+SELinux Extended File Attributes (xattrs) being placed on files created by applications running on
+the compute node, which may not be supported by the destination file system (e.g. Lustre).
+
+Depending on the configuration of `dcp`, there may be an attempt to copy these xattrs. You may need
+to disable this by using `dcp --xattrs none` to avoid errors. For example, the `command` in the
+`nnf-dm-config` config map or `dcpOptions` in the [DataMovementCreateRequest
+API](copy-offload-api.html#datamovement.DataMovementCreateRequest) could be used to set this
+option.
+
+See the [`dcp` documentation](https://mpifileutils.readthedocs.io/en/latest/dcp.1.html) for more
+information.

docs/guides/index.md

@@ -12,6 +12,7 @@
 
 * [Storage Profiles](storage-profiles/readme.md)
 * [Data Movement Configuration](data-movement/readme.md)
+* [Copy Offload API](data-movement/copy-offload-api.html)
 * [Lustre External MGT](external-mgs/readme.md)
 
 ## NNF User Containers

docs/guides/initial-setup/readme.md

@@ -52,7 +52,7 @@ Installation of Kubernetes (k8s) nodes proceeds by installing k8s components ont
 Webhooks require the Jetstack `cert-manager`. Installation is shown below.
 
 ```bash
-export certver="v1.7.0"
+export certver="v1.13.1"
 # Required for webhooks
 kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/"$certver"/cert-manager.yaml
 ```
@@ -63,15 +63,14 @@ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/"$ce
 | :------------------------------| :-------------------- |
 | Generic Kubernetes Worker Node | cray.nnf.manager=true |
 | Rabbit Node                    | cray.nnf.node=true    |
-|                                | cray.nnf.x-name=$NODE |
 
 ### Kubernetes Node Taints
 
 | Node Type                      | Node Label                    |
 | :------------------------------| :---------------------------- |
 | Rabbit Node                    | cray.nnf.node=true:NoSchedule |
 
-See [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). The [NearNodeFlash/nnf-deploy/init.sh script](https://github.com/NearNodeFlash/nnf-deploy/blob/master/init.sh) provides examples of labeling and tainting k8s nodes for use with Rabbit.
+See [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). The [`nnf-deploy init`](https://github.com/NearNodeFlash/nnf-deploy) command provides examples of labeling and tainting k8s nodes for use with Rabbit.
 
 ## Rabbit System Configuration
 
@@ -95,7 +94,7 @@ Here is an example `SystemConfiguration`:
 | storageNodes[].computeAccess | List of {slot, compute name} elements that indicate physical slot index that the named compute node is attached to |
 
 ```yaml
-apiVersion: dws.cray.hpe.com/v1alpha1
+apiVersion: dataworkflowservices.github.io/v1alpha2
 kind: SystemConfiguration
 metadata:
   name: default
@@ -106,6 +105,9 @@ spec:
   - name: compute-02
   - name: compute-03
   - name: compute-04
+  ports:
+  - 5000-5999
+  portsCooldownInSeconds: 0
   storageNodes:
   - computesAccess:
     - index: 0

docs/repo-guides/release-nnf-sw/readme.md

@@ -18,13 +18,15 @@ part of `nnf-deploy`, but it should match the version number of `nnf-deploy`. Re
 other components.
 
 - [NearNodeFlash/nnf-deploy](https://github.com/NearNodeFlash/nnf-deploy)
+
     - [DataWorkflowServices/dws](https://github.com/DataWorkflowServices/dws)
-    - [NearNodeFlash/lustre-fs-operator](https://github.com/NearNodeFlash/lustre-fs-operator)
     - [HewlettPackard/lustre-csi-driver](https://github.com/HewlettPackard/lustre-csi-driver)
+    - [NearNodeFlash/lustre-fs-operator](https://github.com/NearNodeFlash/lustre-fs-operator)
     - [NearNodeFlash/nnf-mfu](https://github.com/NearNodeFlash/nnf-mfu)
     - [NearNodeFlash/nnf-sos](https://github.com/NearNodeFlash/nnf-sos)
     - [NearNodeFlash/nnf-dm](https://github.com/NearNodeFlash/nnf-dm)
     - [NearNodeFlash/nnf-integration-test](https://github.com/NearNodeFlash/nnf-integration-test)
+
 - [NearNodeFlash/NearNodeFlash.github.io](https://github.com/NearNodeFlash/NearNodeFlash.github.io)
 
 [nnf-ec](https://github.com/NearNodeFlash/nnf-ec) is vendored in as part of `nnf-sos` and does not
@@ -33,7 +35,7 @@ need to be released separately.
 ## Primer
 
 This document is based on the process set forth by the [DataWorkflowServices Release
-Process](https://dataworkflowservices.github.io/v0.0.1/repo-guides/create-a-release/readme/).
+Process](https://dataworkflowservices.github.io/v0.0.2/repo-guides/create-a-release/readme/).
 Please read that as a background for this document before going any further.
 
 ## Requirements
@@ -86,97 +88,120 @@ just an example.
 
 5. Perform repo-specific updates:
 
-    1. For `lustre-csi-driver` and `lustre-fs-operator`, there are additional files that need to
+    1. For `lustre-csi-driver`, `lustre-fs-operator`, `dws`, `nnf-sos`, and `nnf-dm` there are additional files that need to
     track the version number as well, which allow them to be installed with `kubectl apply -k`.
 
-        1. For `lustre-fs-operator`, update `config/manager/kustomization.yaml` with the correct
-        version.
-
-        2. For `lustre-csi-driver`, update `deploy/kubernetes/base/kustomization.yaml` and
-        `charts/lustre-csi-driver/values.yaml` with the correct version.
-
-    2. If `nnf-mfu` was updated, multiple references in `nnf-dm` will need to be updated with the
-    new version number for the `nnf-mfu` image:
-
-        1. In `Dockerfile` and `Makefile`, replace `NNFMU_VERSION` with the new version
-
-        2. In `config/manager/kustomization.yaml`, replace `nnf-mfu`'s `newTag: <X.Y.Z>`
+    |Repo                 |Update|
+    |---------------------|------|
+    |`nnf-mfu`            |The new version of `nnf-mfu` is referenced by the `NNFMFU` variable in several places:<br><br>`nnf-sos`<br>1. `Makefile` replace `NNFMFU` with `nnf-mfu's` tag.<br><br>`nnf-dm`<br>1. In `Dockerfile` and `Makefile`, replace `NNFMU_VERSION` with the new version.<br>2. In `config/manager/kustomization.yaml`, replace `nnf-mfu`'s `newTag: <X.Y.Z>.`|
+    |`lustre-fs-operator` |update `config/manager/kustomization.yaml` with the correct version.|
+    |`dws`                |update `config/manager/kustomization.yaml` with the correct version.|
+    |`nnf-sos`            |update `config/manager/kustomization.yaml` with the correct version.|
+    |`nnf-dm`             |update `config/manager/kustomization.yaml` with the correct version.|
+    |`lustre-csi-driver`  |update `deploy/kubernetes/base/kustomization.yaml` and `charts/lustre-csi-driver/values.yaml` with the correct version.|
 
-6. Create a Pull Request from your branch and **target the release branch**.  When merging the Pull
+6. **Target the `releases/v0` branch** with a Pull Request from your branch.  When merging the Pull
 Request, **you must use a Merge Commit.**
 
     !!! note
-
-        **Do not** Rebase or Squash! Those actions will remove the records that Git uses to
+        **Do not** Rebase or Squash! Those actions remove the records that Git uses to
         determine which commits have been merged, and then when the next release is created Git will
         treat everything like a conflict. Additionally, this will cause auto-generated release notes
         to include the previous release.
 
-7. Once merged, update the release branch locally and then create an annotated tag:
+7. Once merged, update the release branch locally and create an annotated tag:
 
     ```shell
     git checkout releases/v0
+    git pull
     git tag -a v0.0.3 -m "Release v0.0.3"
     git push origin --tags
     ```
 
-8. Now that there is a tag, a release can be created via the [GitHub CLI](https://cli.github.com/).
+8. Now that a tag exists, a release can be created via the [GitHub CLI](https://cli.github.com/).
    Alternatively, use the [Web UI](https://github.com/NearNodeFlash/nnf-dm/releases/new).
 
     ```bash
     gh release create --generate-notes --verify-tag -p v0.0.3 -t "Release v0.0.3"
     ```
 
-9. Repeat this process for each remaining component.
+9. GOTO Step 1 and repeat this process for each remaining component.
 
 ## Release `nnf-deploy`
 
-Once the individual components are released, we need to update the submodules and
-`config/repositories.yaml` in the **master** branch before we start on the release branch. This
-makes sure that everything is now current on master.
+Once the individual components are released, we need to update the submodules
+in `nnf-deploy's` `master` branch before we create the release branch. This ensures
+that everything is current on `master` for `nnf-deploy`.
 
-1. Update the submodules on master:
+1. Update the submodules for `nnf-deploy` on master:
 
     ```shell
+    cd nnf-deploy
     git checkout master
     git pull
-    ./update.sh
+    git submodule foreach git checkout master
+    git submodule foreach git pull
     ```
 
-2. Update `config/repositories.yaml` and update the referenced versions for:
-
-    1. `lustre-csi-driver`
+2. Create a branch to capture the submodule changes for the PR to `master`
 
-    2. `lustre-fs-operator`
-
-    3. `nnf-mfu`
+    ```shell
+    git checkout -b update-submodules
+    ```
 
 3. Commit the changes and open a Pull Request against the `master` branch.
 
-4. Once merged, follow steps 1-3 from the previous section to update the release branch with master.
+4. Once merged, follow steps 1-3 from the previous section to create a release branch off of `releases/v0` and
+   update it with changes from `master`.
 
-5. There will be conflicts on the submodules after step 3. This is expected. We will update the
+5. There will be conflicts for the submodules after step 3. **This is expected.** Update the
    submodules to the new tags and then commit the changes.  If each tag was committed properly, the
    following command can do this for you:
 
     ```shell
     git submodule foreach 'git checkout `git describe --match="v*" HEAD`'
     ```
 
-    Verify that each submodule is now at the proper tagged version.
+6. Verify that each submodule is now at the proper tagged version.
+
+    ```shell
+    git submodule
+    ```
+
+7. Do a `git add` for each of the submodules.
+
+8. Update `config/repositories.yaml` with the referenced versions for:
+
+    1. `lustre-csi-driver`
+    2. `lustre-fs-operator`
+    3. `nnf-mfu`  (Search for NNFMFU_VERSION)
+
+9. Tidy and make `nnf-deploy` to avoid embarrassment.
 
     ```shell
-    git submodule status
+    go mod tidy
+    make
     ```
 
-6. Do a `git add` for each of the submodules.
+10. Do another `git add` for any changes, particularly `go.mod` and/or `go.sum`.
+
+11. Verify that `git status` is happy with `nnf-deploy` and then finalize the merge
+    from master by with a `git commit`.
+
+12. Follow steps 6-8 from the previous section to finalize the release of `nnf-deploy`.
+
+**The software is now released!**
+
+## Clone a release
 
-7. Run `go mod tidy` and then `make`. Do another `git add` for any changes, particularly`go.mod`
-and/or `go.sum`.
+The follow commands clone release `v0.0.7` into `nnf-deploy-v0.0.7`
 
-8. Verify that `git status` is happy with `nnf-deploy` and then finalize the merge from master by
-   doing a `git commit`.
+```shell
+export NNF_VERSION=v0.0.7
 
-9. Follow steps 6-8 from the previous section to finalize the release of `nnf-deploy`.
+git clone --recurse-submodules git@github.com:NearNodeFlash/nnf-deploy nnf-deploy-$NNF_VERSION
+cd nnf-deploy-$NNF_VERSION
+git -c advice.detachedHead=false checkout $NNF_VERSION --recurse-submodules
 
-The software is now released!
+git submodule status
+```