-
Notifications
You must be signed in to change notification settings - Fork 4
Add NnfContainerData documentation #307
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -205,6 +205,83 @@ New NnfContainerProfile resources may be restricted to a specific user ID or gro | |
| matching user ID or group ID will be allowed to use that profile . If the profile specifies both of | ||
| these IDs, then the Workflow resource must match both of them. | ||
|
|
||
| #### External Container Creation | ||
|
|
||
| By default a container profile creates Kubernetes `Job` or `MPIJob` resources to run user containers | ||
| on the Rabbit nodes. Setting `createContainer: false` in the profile disables this behaviour, allowing | ||
| an external orchestrator to manage the container lifecycle. All the normal PreRun storage setup still | ||
| takes place — `NnfAccess` resources are created, NNF filesystems are mounted on the Rabbit nodes — | ||
| but **no pods are launched by NNF**. Instead, the NNF software creates an `NnfContainerData` resource | ||
| that records the mount-path information for each storage volume. | ||
|
|
||
| This is useful when an external orchestrator is responsible for driving the actual container lifecycle, | ||
| but still relies on NNF to provision and mount the storage. | ||
|
|
||
| ```yaml | ||
| apiVersion: nnf.cray.hpe.com/v1alpha7 | ||
| kind: NnfContainerProfile | ||
| metadata: | ||
| name: example-profile | ||
| namespace: nnf-system | ||
| data: | ||
| createContainer: false # do not launch pods; create NnfContainerData instead | ||
| storages: | ||
| - name: DW_JOB_local_storage | ||
| optional: false | ||
| spec: | ||
| containers: | ||
| - name: placeholder # spec is still required for validation | ||
| image: "none" | ||
| ``` | ||
|
|
||
| The workflow advances through PreRun, PostRun, and Teardown without blocking on pod lifecycle events. | ||
| The resulting `NnfContainerData` resource is deleted during Teardown alongside any other container | ||
| resources. | ||
|
|
||
| ##### NnfContainerData | ||
|
|
||
| The `NnfContainerData` resource is created in the workflow's namespace during PreRun. It describes | ||
| each NNF storage volume that has been mounted on the Rabbit node, giving the external orchestrator | ||
| the information it needs to attach its own containers to the prepared mounts. | ||
|
|
||
| Given a workflow with the following directives: | ||
|
|
||
| ```bash | ||
| #DW jobdw name=my-gfs2 type=gfs2 capacity=100GB | ||
| #DW persistentdw name=my-persistent | ||
| #DW container name=my-container profile=red-rock-slushy-data-only DW_JOB_local_storage=my-gfs2 DW_PERSISTENT_shared_storage=my-persistent | ||
| ``` | ||
|
|
||
| The `NnfContainerData` resource created during PreRun would look like this: | ||
|
|
||
| ```yaml | ||
| apiVersion: nnf.cray.hpe.com/v1alpha11 | ||
| kind: NnfContainerData | ||
| metadata: | ||
| name: my-workflow-2 | ||
| namespace: default | ||
| data: | ||
| volumes: | ||
| - name: local_storage | ||
| command: jobdw | ||
| directiveIndex: 0 | ||
| mountPath: /mnt/nnf/3e92c060-ca0e-4ddb-905b-3d24137cbff4-0 | ||
| - name: shared_storage | ||
| command: persistentdw | ||
| directiveIndex: 1 | ||
| mountPath: /mnt/nnf/3e92c060-ca0e-4ddb-905b-3d24137cbff4-1 | ||
| ``` | ||
|
|
||
| | Field | Type | Description | | ||
| |---|---|---| | ||
| | `data.volumes[].name` | string | Storage name as defined in the `NnfContainerProfile` (e.g. `local_storage`) | | ||
| | `data.volumes[].command` | string | Directive command that created the storage: `jobdw` or `persistentdw` | | ||
| | `data.volumes[].directiveIndex` | int | Zero-based index of the `#DW` directive in the workflow that created this storage | | ||
| | `data.volumes[].mountPath` | string | Absolute path on the Rabbit node where the storage is mounted | | ||
|
|
||
| The external orchestrator can query the resource by workflow name and namespace to discover where | ||
| each storage volume is mounted: | ||
|
|
||
| ## Creating a Container Workflow | ||
|
|
||
| The user's workflow will specify the name of the `NnfContainerProfile` in a DW directive. If the | ||
|
|
@@ -215,8 +292,7 @@ with the `profile` parameter. | |
| #DW container profile=red-rock-slushy [...] | ||
| ``` | ||
|
|
||
| Furthermore, to set the container storages for the workflow, storage parameters must also be | ||
| supplied in the workflow. This is done using the `<storage_name>` (see [Container | ||
| Furthermore, to set the container storages for the workflow, storage parameters must also be supplied in the workflow. This is done using the `<storage_name>` (see [Container | ||
| Storages](#container-storages)) and setting it to the name of a storage directive that defines an | ||
| NNF filesystem. That storage directive must already exist as part of another workflow (e.g. | ||
| persistent storage) or it can be supplied in the same workflow as the container. For global lustre, | ||
|
|
@@ -275,13 +351,16 @@ overview of the container-related behavior that occurs: | |
| - Proposal: Verify [storages](#container-storages) are provided according to the container profile. | ||
| - Setup: If applicable, [request ports](#container-ports) from NnfPortManager. | ||
| - DataIn: No container related activity. | ||
| - PreRun: Appropriate `MPIJob` or `Job(s)` are created for the workflow. In turn, user containers | ||
| are created and launched by Kubernetes. Containers are expected to start in this state. | ||
| - PostRun: Once in PostRun, user containers are expected to complete (non-zero exit) | ||
| successfully. | ||
| - PreRun: If `createContainer: true` (default), appropriate `MPIJob` or `Job(s)` are created for | ||
| the workflow. In turn, user containers are created and launched by Kubernetes. Containers are | ||
| expected to start in this state. If `createContainer: false`, no pods are launched; instead an | ||
| [`NnfContainerData`](#nnfcontainerdata) resource is created that records the storage mount paths. | ||
| - PostRun: If `createContainer: true`, user containers are expected to complete (non-zero exit) | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. (non-zero exit) would indicate the container failed, wouldn't it? |
||
| successfully. If `createContainer: false`, the NNF code does not coordinate with the external | ||
| container launcher. | ||
| - DataOut: No container related activity. | ||
| - Teardown: Ports are released; `MPIJob` or `Job(s)` are deleted, which in turn deletes the user | ||
| containers. | ||
| containers. Any `NnfContainerData` resource is also deleted. | ||
|
|
||
| The two main states of a container workflow (i.e. PreRun, PostRun) are discussed further in the | ||
| following sections. | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.