Skip to content

Commit 5acb661

Browse files
bdevcichbdevcich
authored
Update Data Movement with Profiles and sshd config
Signed-off-by: Blake Devcich <blake.devcich@hpe.com>
1 parent 5b115ea commit 5acb661

1 file changed

Lines changed: 40 additions & 83 deletions

File tree

‎docs/guides/data-movement/readme.md‎

Lines changed: 40 additions & 83 deletions
Original file line numberDiff line numberDiff line change
@@ -3,115 +3,72 @@ authors: Blake Devcich <blake.devcich@hpe.com>
33
categories: provisioning
44
---
55

6-
# Data Movement Configuration
6+
# Data Movement Overview
7+
8+
## Configuration
79

810
Data Movement can be configured in multiple ways:
911

10-
1. Server side
12+
1. Server side (`NnfDataMovementProfile`)
1113
2. Per Copy Offload API Request arguments
1214

13-
The first method is a "global" configuration - it affects all data movement operations. The second
14-
is done per the Copy Offload API, which allows for some configuration on a per-case basis, but is
15-
limited in scope. Both methods are meant to work in tandem.
16-
17-
## Server Side ConfigMap
18-
19-
The server side configuration is done via the `nnf-dm-config` config map:
20-
21-
```bash
22-
kubectl -n nnf-dm-system get configmap nnf-dm-config
23-
```
24-
25-
The config map allows you to configure the following:
26-
27-
|Setting|Description|
28-
|-------|-----------|
29-
|slots|The number of slots specified in the MPI hostfile. A value less than 1 disables the use of slots in the hostfile.|
30-
|maxSlots|The number of max_slots specified in the MPI hostfile. A value less than 1 disables the use of max_slots in the hostfile.|
31-
|command|The full command to execute data movement. More detail in the following section.|
32-
|progressIntervalSeconds|interval to collect the progress data from the `dcp` command.|
33-
34-
### `command`
35-
36-
The full data movement `command` can be set here. By default, Data Movement uses `mpirun` to run
37-
`dcp` to perform the data movement. Changing the `command` is useful for tweaking `mpirun` or `dcp` options or to
38-
replace the command with something that can aid in debugging (e.g. `hostname`).
39-
40-
`mpirun` uses hostfiles to list the hosts to launch `dcp` on. This hostfile is created for each Data
41-
Movement operation, and it uses the config map to set the `slots` and `maxSlots` for each host (i.e. NNF
42-
node) in the hostfile. The number of `slots`/`maxSlots` is the same for every host in the hostfile.
43-
44-
Additionally, Data Movement uses substitution to fill in dynamic information for each Data Movement
45-
operation. Each of these **must** be present in the command for Data Movement to work properly when
46-
using `mpirun` and `dcp`:
15+
The first method is a "global" configuration - it affects all data movement operations that use a
16+
particular `NnfDataMovementProfile` (or the default). The second is done per the Copy Offload API,
17+
which allows for some configuration on a per-case basis, but is limited in scope. Both methods are
18+
meant to work in tandem.
4719

48-
|VAR|Description|
49-
|---|-----------|
50-
|`$HOSTFILE`|hostfile that is created and used for mpirun.|
51-
|`$UID`|User ID that is inherited from the Workflow.|
52-
|`$GID`|Group ID that is inherited from the Workflow.|
53-
|`$SRC`|source for the data movement.|
54-
|`$DEST`|destination for the data movement.|
20+
### Data Movement Profiles
5521

56-
By default, the command will look something like the following. Please see the config map itself for
57-
the most up to date default command:
22+
The server side configuration is controlled by creating `NnfDataMovementProfiles` resources in
23+
Kubernetes. These work similar to `NnfStorageProfiles`. See [here](../storage-profiles/readme.md)
24+
for understanding how to use profiles, set a default, etc.
5825

59-
```bash
60-
mpirun --allow-run-as-root --hostfile $HOSTFILE dcp --progress 1 --uid $UID --gid $GID $SRC $DEST
61-
```
26+
For an in-depth understanding of the capabilities offered by Data Movement profiles, we recommend
27+
referring to the following resources:
6228

63-
### Profiles
29+
- [Type definition](https://github.com/NearNodeFlash/nnf-sos/blob/master/api/v1alpha1/nnfdatamovementprofile_types.go#L27) for `NnfDataMovementProfile`
30+
- [Sample](https://github.com/NearNodeFlash/nnf-sos/blob/master/config/samples/nnf_v1alpha1_nnfdatamovementprofile.yaml) for `NnfDataMovementProfile`
31+
- [Online Examples](https://github.com/NearNodeFlash/nnf-sos/blob/master/config/examples/nnf_v1alpha1_nnfdatamovementprofile.yaml) for `NnfDataMovementProfile`
6432

65-
Profiles can be specified in the in the `nnf-dm-config` config map. Users are able to select a
66-
profile using #DW directives (e.g .`copy_in profile=my-dm-profile`) and the Copy Offload API. If no
67-
profile is specified, the `default` profile is used. This default profile must exist in the config
68-
map.
69-
70-
`slots`, `maxSlots`, and `command` can be stored in Data Movement profiles. These profiles are
71-
available to quickly switch between different settings for a particular workflow.
72-
73-
Example profiles:
74-
75-
```yaml
76-
profiles:
77-
default:
78-
slots: 8
79-
maxSlots: 0
80-
command: mpirun --allow-run-as-root --hostfile $HOSTFILE dcp --progress 1 --uid $UID --gid $GID $SRC $DEST
81-
no-xattrs:
82-
slots: 8
83-
maxSlots: 0
84-
command: mpirun --allow-run-as-root --hostfile $HOSTFILE dcp --progress 1 --xattrs none --uid $UID --gid $GID $SRC $DEST
85-
```
86-
87-
## Copy Offload API Daemon
33+
### Copy Offload API Daemon
8834

8935
The `CreateRequest` API call that is used to create Data Movement with the Copy Offload API has some
90-
options to allow a user to specify some options for that particular Data Movement. These settings
91-
are on a per-request basis.
92-
93-
The Copy Offload API requires the `nnf-dm` daemon to be running on the compute node. This daemon may be configured to run full-time, or it may be left in a disabled state if the WLM is expected to run it only when a user requests it. See [Compute Daemons](../compute-daemons/readme.md) for the systemd service configuration of the daemon. See `RequiredDaemons` in [Directive Breakdown](../directive-breakdown/readme.md) for a description of how the user may request the daemon, in the case where the WLM will run it only on demand.
36+
options to allow a user to specify some options for that particular Data Movement operation. These
37+
settings are on a per-request basis. These supplement the configuration in the
38+
`NnfDataMovementProfile`.
9439

95-
If the WLM is running the `nnf-dm` daemon only on demand, then the user can request that the daemon be running for their job by specifying `requires=copy-offload` in their `DW` directive. The following is an example:
96-
97-
```bash
98-
#DW jobdw type=xfs capacity=1GB name=stg1 requires=copy-offload
99-
```
40+
The Copy Offload API requires the `nnf-dm` daemon to be running on the compute node. This daemon may
41+
be configured to run full-time, or it may be left in a disabled state if the WLM is expected to run
42+
it only when a user requests it. See [Compute Daemons](../compute-daemons/readme.md) for the systemd
43+
service configuration of the daemon. See `RequiredDaemons` in [Directive
44+
Breakdown](../directive-breakdown/readme.md) for a description of how the user may request the
45+
daemon in the case where the WLM will run it only on demand.
10046

10147
See the [DataMovementCreateRequest API](copy-offload-api.html#datamovement.DataMovementCreateRequest)
10248
definition for what can be configured.
10349

104-
## SELinux and Data Movement
50+
### SELinux and Data Movement
10551

10652
Careful consideration must be taken when enabling SELinux on compute nodes. Doing so will result in
10753
SELinux Extended File Attributes (xattrs) being placed on files created by applications running on
10854
the compute node, which may not be supported by the destination file system (e.g. Lustre).
10955

11056
Depending on the configuration of `dcp`, there may be an attempt to copy these xattrs. You may need
11157
to disable this by using `dcp --xattrs none` to avoid errors. For example, the `command` in the
112-
`nnf-dm-config` config map or `dcpOptions` in the [DataMovementCreateRequest
58+
`NnfDataMovementProfile` or `dcpOptions` in the [DataMovementCreateRequest
11359
API](copy-offload-api.html#datamovement.DataMovementCreateRequest) could be used to set this
11460
option.
11561

11662
See the [`dcp` documentation](https://mpifileutils.readthedocs.io/en/latest/dcp.1.html) for more
11763
information.
64+
65+
### `sshd` Configuration for Data Movement Workers
66+
67+
The `nnf-dm-worker-*` pods run `sshd` in order to listen for `mpirun` jobs to perform data movement.
68+
The number of simultaneous connections is limited via the sshd configuration (i.e. `MaxStartups`).
69+
**If you see error messages in Data Movement where mpirun cannot communicate with target nodes,
70+
and you have ruled out any networking issues, this may be due to sshd configuration.** `sshd` still
71+
start rejecting connections once the limit is reached.
72+
73+
The `sshd_config` is stored in the `nnf-dm-worker-config` `ConfigMap` so that it can be changed on
74+
a running system without needing to roll new images. This also enables site-specific configuration.

0 commit comments

Comments
 (0)