Skip to content

Commit a12b221

Browse files
bdevcichbdevcich
committed
Containers RFC Update
* Added detail to main proposal regarding completion logic for containers - how do we handle success/fail/timeouts? * Added section for Container Assignment * Added communication section detailing Rabbit-to-Rabbit and Compute-to-Rabbit Communication * Use proper prefixes in examples (DW_JOB, DW_PERSISTENT) Signed-off-by: Blake Devcich <blake.devcich@hpe.com>
1 parent 3e65ee5 commit a12b221

1 file changed

Lines changed: 163 additions & 87 deletions

File tree

‎docs/rfcs/0002/readme.md‎

Lines changed: 163 additions & 87 deletions
Original file line numberDiff line numberDiff line change
@@ -2,32 +2,29 @@
22
authors: Nate Thornton <nate.thornton@hpe.com>
33
state: discussion
44
---
5-
Rabbit storage for containerized applications
6-
=============================================
5+
# Rabbit storage for containerized applications
76

87
For Rabbit to provide storage to a containerized application there needs to be _some_ mechanism. The remainder of this RFC proposes that mechanism.
98

10-
Actors
11-
------
9+
## Actors
1210

13-
There are several different actors involved
11+
There are several actors involved:
1412

1513
- The AUTHOR of the containerized application
1614
- The ADMINISTRATOR who works with the author to determine the application requirements for execution
17-
- The USER who intends to to use the application using the 'container' directive in their job specification
15+
- The USER who intends to use the application using the 'container' directive in their job specification
1816
- The RABBIT software that interprets the #DWs and starts the container during execution of the job
1917

20-
There are multiple relationships between the actors
18+
There are multiple relationships between the actors:
2119

2220
- AUTHOR to ADMINISTRATOR: The author tells the administrator how their application is executed and the NNF storage requirements.
2321
- Between the AUTHOR and USER: The application expects certain storage, and the #DW must meet those expectations.
2422
- ADMINISTRATOR to RABBIT: Admin tells Rabbit how to run the containerized application with the required storage.
2523
- Between USER and RABBIT: User provides the #DW container directive in the job specification. Rabbit validates and interprets the directive.
2624

27-
Proposal
28-
--------
25+
## Proposal
2926

30-
The proposal below might take a couple of read-throughs; I've also added a concrete example afterward that might help.
27+
The proposal below outlines the high level behavior of running containers in a workflow:
3128

3229
1. The AUTHOR writes their application expecting NNF Storage at specific locations. For each storage requirement, they define:
3330
1. a unique name for the storage which can be referenced in the 'container' directive
@@ -37,21 +34,84 @@ The proposal below might take a couple of read-throughs; I've also added a concr
3734
2. The AUTHOR works with the ADMINISTRATOR to define:
3835
1. a unique name for the program to be referred by USER
3936
2. the pod template specification for executing their program
40-
3. the NNF storage requirements described above.
41-
3. The ADMINISTRATOR creates a corresponding _NNF Container Profile_ custom kubernetes resource with the necessary NNF storage requirements and pod specification as described by the AUTHOR
42-
4. The USER who desires to use the application works with the AUTHOR and the related NNF Container Profile to understand the storage requirements.
43-
5. The USER submits a WLM job with the #DW container fields populated
44-
6. WLM runs the job and drives the job through the following stages...
45-
1. Proposal: RABBIT validates the #DW container directive by comparing the supplied values to what is listed in the NNF Container Profile. If the USER fails to meet the requirements, the job fails.
46-
2. Pre-run: RABBIT software will:
47-
1. create a config map reflecting the storage requirements and any runtime parameters; this is provided to the container at the volume mount named "nnf-config", if specified.
48-
2. duplicate the pod template specification from the Container Profile and patches the necessary Volumes and the config map. The spec is used as the basis for starting the necessary pods and containers.
49-
3. The containerized application executes. The expected mounts are available per the requirements and celebration occurs.
50-
51-
Example
52-
-------
53-
54-
Say I authored a simple application, `foo`, that requires Rabbit local GFS2 storage and a persistent Lustre storage volume. As the author, my program is coded to expect the GFS2 volume is mounted at `/foo/local` and the Lustre volume is mounted at `/foo/persistent`. In this case, the storages are not optional, so they are defined as such in the NNF Container Profile.
37+
3. the NNF storage requirements described above.
38+
3. The ADMINISTRATOR creates a corresponding _NNF Container Profile_ Kubernetes custom resource with the necessary NNF storage requirements and pod specification as described by the AUTHOR
39+
4. The USER who desires to use the application works with the AUTHOR and the related NNF Container Profile to understand the storage requirements
40+
5. The USER submits a WLM job with the #DW container directive variables populated
41+
6. WLM runs the workflow and drives it through the following stages...
42+
1. `Proposal`: RABBIT validates the #DW container directive by comparing the supplied values to those listed in the NNF Container Profile. If the workflow fails to meet the requirements, the job fails
43+
2. `PreRun`: RABBIT software:
44+
1. creates a config map reflecting the storage requirements and any runtime parameters; this is provided to the container at the volume mount named `nnf-config`, if specified
45+
2. duplicates the pod template specification from the Container Profile and patches the necessary Volumes and the config map. The spec is used as the basis for starting the necessary pods and containers
46+
3. The containerized application(s) executes. The expected mounts are available per the requirements and celebration occurs. The pods continue to run until:
47+
1. a pod completes successfully (any failed pods will be retried)
48+
2. the max number of pod retries is hit (indicating failure on all retry attempts)
49+
1. Note: retry limit is non-optional per Kubernetes configuration
50+
2. If retries are not desired, this number could be set to 0 to disable any retry attempts
51+
4. `PostRun`: RABBIT software:
52+
1. marks the stage as `Ready` if the pods have all completed successfully. This includes a successful retry after preceding failures
53+
2. starts a timer for any running pods. Once the timeout is hit, the pods will be killed and the workflow will indicate failure
54+
3. leaves all pods around for log inspection
55+
56+
### Container Assignment to Rabbit Nodes
57+
58+
During `Proposal`, the USER must assign compute nodes for the container workflow. The assigned compute nodes determine which Rabbit nodes run the containers.
59+
60+
### Communication Details
61+
62+
The following subsections outline the proposed communication between the Rabbit nodes themselves and the Compute nodes.
63+
64+
#### Rabbit-to-Rabbit Communication
65+
66+
Each rabbit node can be reached via `<hostname>.<subdomain>` using DNS. The hostname is a combination of the workflow name and Rabbit node name. The workflow name is used for the subdomain.
67+
68+
For example, a workflow name of `foo` that targets `rabbit-node2` would be `foo-rabbit-node2.foo`.
69+
70+
Environment variables are provided to the container and ConfigMap for each rabbit that is targeted by the container workflow:
71+
72+
```shell
73+
NNF_CONTAINER_NODES=foo-rabbit-node2,foo-rabbit-node3
74+
NNF_CONTAINER_SUBDOMAIN=foo
75+
NNF_CONTAINER_DOMAIN=default.svc.cluster.local
76+
```
77+
78+
```yaml
79+
kind: ConfigMap
80+
apiVersion: v1
81+
data:
82+
nnfContainerNodes:
83+
- foo-rabbit-node2
84+
- foo-rabbit-node3
85+
nnfContainerSubdomain: foo
86+
nnfContainerDomain: default.svc.cluster.local
87+
```
88+
89+
DNS can then be used to communicate with other Rabbit containers. The FQDN for the container running on rabbit-node2 is `foo-rabbit-node2.foo.default.svc.cluster.local`.
90+
91+
#### Compute-to-Rabbit Communication
92+
93+
For Compute to Rabbit communication, the proposal is to use an open port between the nodes, so the applications could communicate using IP protocol. The port number would be assigned by the Rabbit software and included in the workflow resource's environmental variables after the Setup state (similar to workflow name & namespace). Flux should provide the port number to the compute application via an environmental variable or command line argument. The containerized application would always see the same port number using the `hostPort`/`containerPort` mapping functionality included in Kubernetes. To clarify, the Rabbit software is picking and managing the ports picked for `hostPort`.
94+
95+
This requires a range of ports to be open in the firewall configuration and specified in the rabbit system configuration. The fewer the number of ports available increases the chances of a port reservation conflict that would fail a workflow.
96+
97+
Example port range definition in the SystemConfiguration:
98+
99+
```yaml
100+
apiVersion: v1
101+
items:
102+
- apiVersion: dws.cray.hpe.com/v1alpha1
103+
kind: SystemConfiguration
104+
name: default
105+
namespace: default
106+
spec:
107+
containerHostPortRangeMin: 30000
108+
containerHostPortRangeMax: 40000
109+
...
110+
```
111+
112+
## Example
113+
114+
Say I authored a simple application, `foo`, that requires Rabbit local GFS2 storage and a persistent Lustre storage volume. As the author, my program is coded to expect the GFS2 volume is mounted at `/foo/local` and the Lustre volume is mounted at `/foo/persistent`
55115

56116
Working with an administrator, my application's storage requirements and pod specification are placed in an NNF Container Profile `foo`:
57117

@@ -62,10 +122,12 @@ metadata:
62122
name: foo
63123
namespace: default
64124
spec:
125+
postRunTimeout: 300
126+
maxRetries: 6
65127
storages:
66-
- name: JOB_DW_foo-local-storage
128+
- name: DW_JOB_foo-local-storage
67129
optional: false
68-
- name: PERSISTENT_DW_foo-persistent-storage
130+
- name: DW_PERSISTENT_foo-persistent-storage
69131
optional: false
70132
template:
71133
metadata:
@@ -84,18 +146,21 @@ spec:
84146
mountPath: /foo/persistent
85147
- name: nnf-config
86148
mountPath: /nnf/config
149+
ports:
150+
- name: compute
151+
containerPort: 80
87152
```
88153

89154
Say Peter wants to use `foo` as part of his job specification. Peter would submit the job with the directives below:
90155

91-
```
156+
```text
92157
#DW jobdw name=my-gfs2 type=gfs2 capacity=1TB
93158
94159
#DW persistentdw name=some-lustre
95160
96161
#DW container name=my-foo profile=foo \
97-
JOB_DW_foo-local-storage=my-gfs2 \
98-
PERSISTENT_DW_foo-persistent-storage=some-lustre
162+
DW_JOB_foo-local-storage=my-gfs2 \
163+
DW_PERSISTENT_foo-persistent-storage=some-lustre
99164
```
100165

101166
Since the NNF Container Profile has specified that both storages are not optional (i.e. `optional: false`), they must both be present in the #DW directives along with the `container` directive. Alternatively, if either was marked as optional (i.e. `optional: true`), it would not be required to be present in the #DW directives and therefore would not be mounted into the container.
@@ -106,76 +171,87 @@ Peter submits the job to the WLM. WLM guides the job through the workflow states
106171
2. Setup: Since there is a jobdw, `my-gfs2`, Rabbit software provisions this storage.
107172
3. Pre-Run:
108173
1. Rabbit software generates a config map that corresponds to the storage requirements and runtime parameters.
109-
```yaml
110-
kind: ConfigMap
111-
apiVersion: v1
112-
metadata:
113-
name: my-job-container-my-foo
114-
data:
115-
JOB_DW_foo-local-storage: type=gfs2 mount-type=indexed-mount
116-
PERSISTENT_DW_foo-persistent-storage: type=lustre mount-type=mount-point
117-
```
118-
2. Rabbit software duplicates the `foo` pod template spec in the NNF Container Profile and fills in the necessary volumes and config map.
119-
```yaml
120-
kind: Pod
121-
apiVersion: v1
122-
metadata:
123-
name: my-job-container-my-foo
124-
template:
125-
metadata:
126-
name: foo
127-
namespace: default
128-
spec:
129-
containers:
130-
# This section unchanged from Container Profile
131-
- name: foo
132-
image: foo:latest
133-
command:
134-
- /foo
135-
volumeMounts:
136-
- name: foo-local-storage
137-
mountPath: /foo/local
138-
- name: foo-persistent-storage
139-
mountPath: /foo/persistent
140-
- name: nnf-config
141-
mountPath: /nnf/config
142174

143-
# volumes added by Rabbit software
144-
volumes:
145-
- name: foo-local-storage
146-
hostPath:
147-
path: /nnf/job/my-job/my-gfs2
148-
- name: foo-persistent-storage
149-
hostPath:
150-
path: /nnf/persistent/some-lustre
151-
- name: nnf-config
152-
configMap:
175+
```yaml
176+
kind: ConfigMap
177+
apiVersion: v1
178+
metadata:
153179
name: my-job-container-my-foo
180+
data:
181+
DW_JOB_foo-local-storage: type=gfs2 mount-type=indexed-mount
182+
DW_PERSISTENT_foo-persistent-storage: type=lustre mount-type=mount-point
183+
...
184+
```
185+
186+
2. Rabbit software creates a pod and duplicates the `foo` pod template spec in the NNF Container Profile and fills in the necessary volumes and config map.
187+
188+
```yaml
189+
kind: Pod
190+
apiVersion: v1
191+
metadata:
192+
name: my-job-container-my-foo
193+
template:
194+
metadata:
195+
name: foo
196+
namespace: default
197+
spec:
198+
containers:
199+
# This section unchanged from Container Profile
200+
- name: foo
201+
image: foo:latest
202+
command:
203+
- /foo
204+
volumeMounts:
205+
- name: foo-local-storage
206+
mountPath: /foo/local
207+
- name: foo-persistent-storage
208+
mountPath: /foo/persistent
209+
- name: nnf-config
210+
mountPath: /nnf/config
211+
ports:
212+
- name: compute
213+
hostPort: 9376 # hostport selected by Rabbit software
214+
containerPort: 80
215+
216+
# volumes added by Rabbit software
217+
volumes:
218+
- name: foo-local-storage
219+
hostPath:
220+
path: /nnf/job/my-job/my-gfs2
221+
- name: foo-persistent-storage
222+
hostPath:
223+
path: /nnf/persistent/some-lustre
224+
- name: nnf-config
225+
configMap:
226+
name: my-job-container-my-foo
227+
228+
# securityContext added by Rabbit software - values will be inherited from the workflow
229+
securityContext:
230+
runAsUser: 1000
231+
runAsGroup: 2000
232+
fsGroup: 2000
233+
```
154234

155-
# securityContext added by Rabbit software - values will be inherited from the workflow
156-
securityContext:
157-
runAsUser: 1000
158-
runAsGroup: 2000
159-
fsGroup: 2000
160-
```
161235
3. Rabbit software starts the pods on Rabbit nodes
236+
4. Post-Run
237+
1. Rabbit waits for all pods to finish (or until timeout is hit)
238+
2. If all pods are successful, Post-Run is marked as `Ready`
239+
3. If any pod is not successful, Post-Run is not marked as `Ready`
162240

163-
Security
164-
--------
241+
## Security
165242

166-
Kubernetes allows for a way to define permissions for a container using a [Security Context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/). This can be seen in the pod template spec above. The user and group IDs will be inherited from the Workflow's spec.
243+
Kubernetes allows for a way to define permissions for a container using a [Security Context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/). This can be seen in the pod template spec above. The user and group IDs are inherited from the Workflow's spec.
167244

168-
Special Note: Indexed-Mount Type
169-
--------------------------------
245+
## Special Note: Indexed-Mount Type
170246

171247
When using a file system like XFS or GFS2, each compute is allocated its own Rabbit volume. The Rabbit software mounts a collection of mount paths with a common prefix and an ending indexed value.
172248

173249
Application AUTHORS must be aware that their desired mount-point really contains a collection of directories, one for each compute node. The mount point type can be known by consulting the config map values.
174250

175-
If we continue the example from above, the `foo` application would expect the foo-local-storage path of `/foo/local` to contain several directories
251+
If we continue the example from above, the `foo` application expects the foo-local-storage path of `/foo/local` to contain several directories
176252

177253
```shell
178-
# ls /foo/local/*
254+
$ ls /foo/local/*
179255
180256
node-0
181257
node-1
@@ -184,7 +260,7 @@ node-2
184260
node-N
185261
```
186262

187-
Node positions are ***not*** absolute locations. WLM could, in theory, select 6 physical compute nodes at physical location 1, 2, 3, 5, 8, 13, which would appear as directories `/node-0` through `/node-5` in the container path.
263+
Node positions are _not_ absolute locations. WLM could, in theory, select 6 physical compute nodes at physical location 1, 2, 3, 5, 8, 13, which would appear as directories `/node-0` through `/node-5` in the container path.
188264

189265
Symlinks will be added to support the physical compute node names. Assuming a compute node hostname of `compute-node-1` from the example above, it would link to `node-0`, `compute-node-2` would link to `node-1`, etc.
190266

0 commit comments

Comments
 (0)