|
| 1 | +--- |
| 2 | +authors: Blake Devcich <blake.devcich@hpe.com> |
| 3 | +categories: provisioning |
| 4 | +--- |
| 5 | + |
| 6 | +# Data Movement Configuration |
| 7 | + |
| 8 | +Data Movement can be configured in multiple ways: |
| 9 | + |
| 10 | +1. Server side |
| 11 | +2. Per Copy Offload API Request arguments |
| 12 | + |
| 13 | +The first method is a "global" configuration - it affects all data movement operations. The second |
| 14 | +is done per the Copy Offload API, which allows for some configuration on a per-case basis, but is |
| 15 | +limited in scope. Both methods are meant to work in tandem. |
| 16 | + |
| 17 | +## Server Side ConfigMap |
| 18 | + |
| 19 | +The server side configuration is done via the `nnf-dm-config` config map: |
| 20 | + |
| 21 | +```bash |
| 22 | +kubectl -n nnf-dm-system get configmap nnf-dm-config |
| 23 | +``` |
| 24 | + |
| 25 | +The config map allows you to configure the following: |
| 26 | + |
| 27 | +|Setting|Description| |
| 28 | +|-------|-----------| |
| 29 | +|slots|The number of slots specified in the MPI hostfile. A value less than 1 disables the use of slots in the hostfile.| |
| 30 | +|maxSlots|The number of max_slots specified in the MPI hostfile. A value less than 1 disables the use of max_slots in the hostfile.| |
| 31 | +|command|The full command to execute data movement. More detail in the following section.| |
| 32 | +|progressIntervalSeconds|interval to collect the progress data from the `dcp` command.| |
| 33 | + |
| 34 | +### `command` |
| 35 | + |
| 36 | +The full data movement `command` can be set here. By default, Data Movement uses `mpirun` to run |
| 37 | +`dcp` to perform the data movement. Changing the `command` is useful for tweaking `mpirun` or `dcp` options or to |
| 38 | +replace the command with something that can aid in debugging (e.g. `hostname`). |
| 39 | + |
| 40 | +`mpirun` uses hostfiles to list the hosts to launch `dcp` on. This hostfile is created for each Data |
| 41 | +Movement operation, and it uses the config map to set the `slots` and `maxSlots` for each host (i.e. NNF |
| 42 | +node) in the hostfile. The number of `slots`/`maxSlots` is the same for every host in the hostfile. |
| 43 | + |
| 44 | +Additionally, Data Movement uses substitution to fill in dynamic information for each Data Movement |
| 45 | +operation. Each of these **must** be present in the command for Data Movement to work properly when |
| 46 | +using `mpirun` and `dcp`: |
| 47 | + |
| 48 | +|VAR|Description| |
| 49 | +|---|-----------| |
| 50 | +|`$HOSTFILE`|hostfile that is created and used for mpirun.| |
| 51 | +|`$UID`|User ID that is inherited from the Workflow.| |
| 52 | +|`$GID`|Group ID that is inherited from the Workflow.| |
| 53 | +|`$SRC`|source for the data movement.| |
| 54 | +|`$DEST`|destination for the data movement.| |
| 55 | + |
| 56 | +By default, the command will look something like the following. Please see the config map itself for |
| 57 | +the most up to date default command: |
| 58 | + |
| 59 | +```bash |
| 60 | +mpirun --allow-run-as-root --hostfile $HOSTFILE dcp --progress 1 --uid $UID --gid $GID $SRC $DEST |
| 61 | +``` |
| 62 | + |
| 63 | +### Profiles |
| 64 | + |
| 65 | +**Note:** This feature is not fully implemented, but is present in the `nnf-dm-config` config map. |
| 66 | +Only the `default` profile is used, and it must be present in the configuration. Once the feature is |
| 67 | +implemented, a user will be able to select a profile using #DW directives and/or the Copy Offload |
| 68 | +API. Right now, users can add additional profiles into the config map, but the only way to use them |
| 69 | +would be to rename one of them to be `default`. |
| 70 | + |
| 71 | +`slots`, `maxSlots`, and `command` can be stored in Data Movement profiles. These profiles are |
| 72 | +available for a quick way to switch between different settings for a particular workflow. |
| 73 | + |
| 74 | +Example profile: |
| 75 | + |
| 76 | +```yaml |
| 77 | +profiles: |
| 78 | + default: |
| 79 | + slots: 8 |
| 80 | + maxSlots: 0 |
| 81 | + command: mpirun --allow-run-as-root --hostfile $HOSTFILE dcp --progress 1 --uid $UID --gid $GID $SRC $DEST |
| 82 | +``` |
| 83 | +
|
| 84 | +## Copy Offload API Daemon |
| 85 | +
|
| 86 | +The `CreateRequest` API call that is used to create Data Movement with the Copy Offload API has some |
| 87 | +options to allow a user to specify some options for that particular Data Movement. These settings |
| 88 | +are on a per-request basis. |
| 89 | + |
| 90 | +See the [CreateRequest API](https://github.com/NearNodeFlash/nnf-dm/blob/master/daemons/compute/Readme.md#create-request) |
| 91 | +definition for what can be configured. |
0 commit comments