Skip to content

Commit e855483

Browse files
bdevcichbdevcich
authored
Merge pull request #73 from NearNodeFlash/release-v0.0.2
Release v0.0.2
2 parents 8ba5e6f + a1826bc commit e855483

8 files changed

Lines changed: 530 additions & 133 deletions

File tree

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,4 @@
11
venv
2+
site
3+
24

‎docs/guides/compute-daemons/readme.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ SERVICE_ACCOUNT=$1
4040
NAMESPACE=$2
4141

4242
kubectl get secret ${SERVICE_ACCOUNT} -n ${NAMESPACE} -o json | jq -Mr '.data.token' | base64 --decode > ./service.token
43-
kubectl get secret ${SERVICE_ACCOUNT} -n ${NAMESPACE} -o json | jq -Mr '.data["ca.crt"]' | base64 -decode > ./service.cert
43+
kubectl get secret ${SERVICE_ACCOUNT} -n ${NAMESPACE} -o json | jq -Mr '.data["ca.crt"]' | base64 --decode > ./service.cert
4444
```
4545

4646
The `service.token` and `service.cert` files must be copied to each compute node, typically in the `/etc/[BINARY-NAME]/` directory

‎docs/guides/ha-cluster/readme.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ Configure the NNF agent with the following parameters:
4646
| `nnf-node-name=[NNF-NODE-NAME]` | Name of the NNF node as it is appears in the System Configuration |
4747
| `api-version=[VERSION]` | The API Version of the NNF Node resource. Defaults to "v1alpha1" |
4848

49-
The token and certificate can be found in the Kubernetes Secrets resource for the nnf-system/nnf-fence-agent ServiceAccount. This provides RBAC rules to limit the fencing agent to only the Kubernetes resources it needs access to.
49+
The token and certificate can be found in the Kubernetes Secrets resource for the nnf-system/nnf-fencing-agent ServiceAccount. This provides RBAC rules to limit the fencing agent to only the Kubernetes resources it needs access to.
5050

5151
For example, setting up the NNF fencing agent on `rabbit-node-1` with a kubernetes service API running at `192.168.0.1:6443` and the service token and certificate copied to `/etc/nnf/fence/`. This needs to be run on one node in the cluster.
5252

‎docs/guides/index.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,9 @@
1212

1313
* [Storage Profiles](storage-profiles/readme.md)
1414
* [Data Movement Configuration](data-movement/readme.md)
15+
16+
## NNF User Containers
17+
18+
* [User Containers](user-containers/readme.md)
19+
20+

‎docs/guides/rbac-for-users/readme.md‎

Lines changed: 72 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,15 @@ authors: Matt Richerson <matthew.richerson@hpe.com>
33
categories: setup
44
---
55

6-
# RBAC for Users
6+
# RBAC: Role-Based Access Control
77

8-
This document shows how to create a kubeconfig file with RBAC set up to restrict access to view only for resources.
8+
RBAC (Role Based Access Control) determines the operations a user or service can perform on a list of Kubernetes resources. RBAC affects everything that interacts with the kube-apiserver (both users and services internal or external to the cluster). More information about RBAC can be found in the Kubernetes [***documentation***](https://kubernetes.io/docs/reference/access-authn-authz/rbac/).
99

10-
## Overview
10+
## RBAC for Users
1111

12-
RBAC (Role Based Access Control) determines the operations a user or service can perform on a list of Kubernetes resources. RBAC affects everything that interacts with the kube-apiserver (both users and services internal or external to the cluster). More information about RBAC can be found in the Kubernetes [***documentation***](https://kubernetes.io/docs/reference/access-authn-authz/rbac/).
12+
This section shows how to create a kubeconfig file with RBAC set up to restrict access to view only for resources.
13+
14+
### Overview
1315

1416
User access to a Kubernetes cluster is defined through a kubeconfig file. This file contains the address of the kube-apiserver as well as the key and certificate for the user. Typically this file is located in `~/.kube/config`. When a kubernetes cluster is created, a config file is generated for the admin that allows unrestricted access to all resources in the cluster. This is the equivalent of `root` on a Linux system.
1517

@@ -19,46 +21,49 @@ The goal of this document is to create a new kubeconfig file that allows view on
1921
- Creating a new kubeconfig file
2022
- Adding RBAC rules for the "hpe" user to allow read access
2123

22-
## Generate a Key and Certificate
24+
### Generate a Key and Certificate
2325

2426
The first step is to create a new key and certificate so that HPE employees can authenticate as the "hpe" user. This will likely be done on one of the master nodes. The `openssl` command needs access to the certificate authority file. This is typically located in `/etc/kubernetes/pki`.
2527

2628
```bash
2729

2830
# make a temporary work space
29-
mkdir /tmp/hpe
30-
cd /tmp/hpe
31+
mkdir /tmp/rabbit
32+
cd /tmp/rabbit
33+
34+
# Create this user
35+
export USERNAME=hpe
3136

3237
# generate a new key
33-
openssl genrsa -out hpe.key 2048
38+
openssl genrsa -out rabbit.key 2048
3439

35-
# create a certificate signing request for the "hpe" user
36-
openssl req -new -key hpe.key -out hpe.csr -subj "/CN=hpe"
40+
# create a certificate signing request for this user
41+
openssl req -new -key rabbit.key -out rabbit.csr -subj "/CN=$USERNAME"
3742

3843
# generate a certificate using the certificate authority on the k8s cluster. This certificate lasts 500 days
39-
openssl x509 -req -in hpe.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out hpe.crt -days 500
44+
openssl x509 -req -in rabbit.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out rabbit.crt -days 500
4045

4146
```
4247

43-
## Create a kubeconfig
48+
### Create a kubeconfig
4449

45-
After the keys have been generated, a new kubeconfig file can be created for the "hpe" user. The admin kubeconfig `/etc/kubernetes/admin.conf` can be used to determine the cluster name kube-apiserver address.
50+
After the keys have been generated, a new kubeconfig file can be created for this user. The admin kubeconfig `/etc/kubernetes/admin.conf` can be used to determine the cluster name kube-apiserver address.
4651

4752
```bash
4853

4954
# create a new kubeconfig with the server information
50-
kubectl config set-cluster {CLUSTER_NAME} --kubeconfig=/tmp/hpe/hpe.conf --server={SERVER_ADDRESS} --certificate-authority=/etc/kubernetes/pki/ca.crt --embed-certs=true
55+
kubectl config set-cluster $CLUSTER_NAME --kubeconfig=/tmp/rabbit/rabbit.conf --server=$SERVER_ADDRESS --certificate-authority=/etc/kubernetes/pki/ca.crt --embed-certs=true
5156

52-
# add the key and cert for the "hpe" user to the config
53-
kubectl config set-credentials hpe --kubeconfig=/tmp/hpe/hpe.conf --client-certificate=/tmp/hpe/hpe.crt --client-key=/tmp/hpe/hpe.key --embed-certs=true
57+
# add the key and cert for this user to the config
58+
kubectl config set-credentials $USERNAME --kubeconfig=/tmp/rabbit/rabbit.conf --client-certificate=/tmp/rabbit/rabbit.crt --client-key=/tmp/rabbit/rabbit.key --embed-certs=true
5459

5560
# add a context
56-
kubectl config set-context hpe-context --kubeconfig=/tmp/hpe/hpe.conf --cluster={CLUSTER_NAME} --user=hpe
61+
kubectl config set-context $USERNAME --kubeconfig=/tmp/rabbit/rabbit.conf --cluster=$CLUSTER_NAME --user=$USERNAME
5762
```
5863

5964
The kubeconfig file should be placed in a location where HPE employees have read access to it.
6065

61-
## Create ClusterRole and ClusterRoleBinding
66+
### Create ClusterRole and ClusterRoleBinding
6267

6368
The next step is to create ClusterRole and ClusterRoleBinding resources. The ClusterRole provided allows viewing all cluster and namespace scoped resources, but disallows creating, deleting, or modifying any resources.
6469

@@ -92,10 +97,58 @@ roleRef:
9297
9398
Both of these resources can be created using the `kubectl apply` command.
9499

95-
## Testing
100+
### Testing
96101

97102
Get, List, Create, Delete, and Modify operations can be tested as the "hpe" user by setting the KUBECONFIG environment variable to use the new kubeconfig file. Get and List should be the only allowed operations. Other operations should fail with a "forbidden" error.
98103

99104
```bash
100105
export KUBECONFIG=/tmp/hpe/hpe.conf
101106
```
107+
108+
## RBAC for Workload Manager (WLM)
109+
110+
**Note** This section assumes the reader has read and understood the steps described above for setting up `RBAC for Users`.
111+
112+
A workload manager (WLM) such as [Flux](https://github.com/flux-framework) or [Slurm](https://slurm.schedmd.com) will interact with [DataWorkflowServices](https://dataworkflowservices.github.io) as a privileged user. RBAC is used to limit the operations that a WLM can perform on a Rabbit system.
113+
114+
The following steps are required to create a user and a role for the WLM. In this case, we're creating a user to be used with the Flux WLM:
115+
116+
- Generate a new key/cert pair for a "flux" user
117+
- Creating a new kubeconfig file
118+
- Adding RBAC rules for the "flux" user to allow appropriate access to the DataWorkflowServices API.
119+
120+
### Generate a Key and Certificate
121+
122+
Generate a key and certificate for our "flux" user, similar to the way we created one for the "hpe" user above. Substitute "flux" in place of "hpe".
123+
124+
### Create a kubeconfig
125+
126+
After the keys have been generated, a new kubeconfig file can be created for the "flux" user, similar to the one for the "hpe" user above. Again, substitute "flux" in place of "hpe".
127+
128+
### Apply the provided ClusterRole and create a ClusterRoleBinding
129+
130+
DataWorkflowServices has already defined the role to be used with WLMs. Simply apply the `workload-manager` ClusterRole from DataWorkflowServices to the system:
131+
132+
```console
133+
kubectl apply -f https://github.com/HewlettPackard/dws/raw/master/config/rbac/workload_manager_role.yaml
134+
```
135+
136+
Create and apply a ClusterRoleBinding to associate the "flux" user with the `workload-manager` ClusterRole:
137+
138+
ClusterRoleBinding
139+
```yaml
140+
apiVersion: rbac.authorization.k8s.io/v1
141+
kind: ClusterRoleBinding
142+
metadata:
143+
name: flux
144+
subjects:
145+
- kind: User
146+
name: flux
147+
apiGroup: rbac.authorization.k8s.io
148+
roleRef:
149+
kind: ClusterRole
150+
name: workload-manager
151+
apiGroup: rbac.authorization.k8s.io
152+
```
153+
154+
The WLM should then use the kubeconfig file associated with this "flux" user to access the DataWorkflowServices API and the Rabbit system.

‎docs/guides/user-containers/readme.md‎

Lines changed: 200 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,200 @@
1+
# NNF User Containers
2+
3+
NNF User Containers are a mechanism to allow user-defined containerized
4+
applications to be run on Rabbit nodes with access to NNF ephemeral and persistent storage.
5+
6+
!!! note
7+
8+
The following is a limited look at User Containers. More content will be
9+
provided after the RFC has been finalized.
10+
11+
## Custom NnfContainerProfile
12+
13+
The author of a containerized application will work with the administrator to
14+
define a pod specification template for the container and to create an
15+
appropriate NnfContainerProfile resource for the container. The image and tag
16+
for the user's container will be specified in the profile.
17+
18+
New NnfContainerProfile resources may be created by copying one of the provided
19+
example profiles from the `nnf-system` namespace. The examples may be found by listing them with `kubectl`:
20+
21+
```console
22+
kubectl get nnfcontainerprofiles -n nnf-system
23+
```
24+
25+
### Workflow Job Specification
26+
27+
The user's workflow will specify the name of the NnfContainerProfile in a DW
28+
directive. If the custom profile is named `red-rock-slushy` then it will be
29+
specified in the "#DW container" directive with the "profile" parameter.
30+
31+
```bash
32+
#DW container profile=red-rock-slushy [...]
33+
```
34+
35+
## Using a Private Container Repository
36+
37+
The user's containerized application may be placed in a private repository. In
38+
this case, the user must define an access token to be used with that repository,
39+
and that token must be made available to the Rabbit's Kubernetes environment
40+
so that it can pull that container from the private repository.
41+
42+
See [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/) in the Kubernetes documentation
43+
for more information.
44+
45+
### About the Example
46+
47+
Each container registry will have its own way of letting its users create tokens to
48+
be used with their repositories. Docker Hub will be used for the private repository in this example, and the user's account on Docker Hub will be "dean".
49+
50+
### Preparing the Private Repository
51+
52+
The user's application container is named "red-rock-slushy". To store this container
53+
on Docker Hub the user must log into docker.com with their browser and click the "Create repository" button to create a repository named "red-rock-slushy", and the user must check the box that marks the repository as private. The repository's name will be displayed as "dean/red-rock-slushy" with a lock icon to show that it is private.
54+
55+
### Create and Push a Container
56+
57+
The user will create their container image in the usual ways, naming it for their private repository and tagging it according to its release.
58+
59+
Prior to pushing images to the repository, the user must complete a one-time login to the Docker registry using the docker command-line tool.
60+
61+
```console
62+
docker login -u dean
63+
```
64+
65+
After completing the login, the user may then push their images to the repository.
66+
67+
```console
68+
docker push dean/red-rock-slushy:v1.0
69+
```
70+
71+
### Generate a Read-Only Token
72+
73+
A read-only token must be generated to allow Kubernetes to pull that container
74+
image from the private repository, because Kubernetes will not be running as
75+
that user. **This token must be given to the administrator, who will use it to create a Kubernetes secret.**
76+
77+
To log in and generate a read-only token to share with the administrator, the user must follow these steps:
78+
79+
- Visit docker.com and log in using their browser.
80+
- Click on the username in the upper right corner.
81+
- Select "Account Settings" and navigate to "Security".
82+
- Click the "New Access Token" button to create a read-only token.
83+
- Keep a copy of the generated token to share with the administrator.
84+
85+
### Store the Read-Only Token as a Kubernetes Secret
86+
87+
The adminstrator must store the user's read-only token as a kubernetes secret. The
88+
secret must be placed in the `default` namespace, which is the same namespace
89+
where the user containers will be run. The secret must include the user's Docker
90+
Hub username and the email address they have associated with that username. In
91+
this case, the secret will be named `readonly-red-rock-slushy`.
92+
93+
```console
94+
$ USER_TOKEN=users-token-text
95+
$ USER_NAME=dean
96+
$ USER_EMAIL=dean@myco.com
97+
$ SECRET_NAME=readonly-red-rock-slushy
98+
$ kubectl create secret docker-registry $SECRET_NAME -n default --docker-server="https://index.docker.io/v1/" --docker-username=$USER_NAME --docker-password=$USER_TOKEN --docker-email=$USER_EMAIL
99+
```
100+
101+
### Add the Secret to the NnfContainerProfile
102+
103+
The administrator must add an `imagePullSecrets` list to the NnfContainerProfile
104+
resource that was created for this user's containerized application.
105+
106+
The following profile shows the placement of the `readonly-red-rock-slushy` secret
107+
which was created in the previous step, and points to the user's
108+
`dean/red-rock-slushy:v1.0` container.
109+
110+
```yaml
111+
apiVersion: nnf.cray.hpe.com/v1alpha1
112+
kind: NnfContainerProfile
113+
metadata:
114+
name: red-rock-slushy
115+
namespace: nnf-system
116+
data:
117+
pinned: false
118+
retryLimit: 6
119+
spec:
120+
imagePullSecrets:
121+
- name: readonly-red-rock-slushy
122+
containers:
123+
- command:
124+
- /users-application
125+
image: dean/red-rock-slushy:v1.0
126+
name: red-rock-app
127+
storages:
128+
- name: DW_JOB_foo_local_storage
129+
optional: false
130+
- name: DW_PERSISTENT_foo_persistent_storage
131+
optional: true
132+
```
133+
134+
Now any user can select this profile in their Workflow by specifying it in a
135+
`#DW container` directive.
136+
137+
```bash
138+
#DW container profile=red-rock-slushy [...]
139+
```
140+
141+
### Using a Private Container Repository for MPI Application Containers
142+
143+
If our user's containerized application instead contains an MPI application,
144+
because perhaps it's a private copy of [nnf-mfu](https://github.com/NearNodeFlash/nnf-mfu),
145+
then the administrator would insert two `imagePullSecrets` lists into the
146+
`mpiSpec` of the NnfContainerProfile for the MPI launcher and the MPI worker.
147+
148+
```yaml
149+
apiVersion: nnf.cray.hpe.com/v1alpha1
150+
kind: NnfContainerProfile
151+
metadata:
152+
name: mpi-red-rock-slushy
153+
namespace: nnf-system
154+
data:
155+
mpiSpec:
156+
mpiImplementation: OpenMPI
157+
mpiReplicaSpecs:
158+
Launcher:
159+
template:
160+
spec:
161+
imagePullSecrets:
162+
- name: readonly-red-rock-slushy
163+
containers:
164+
- command:
165+
- mpirun
166+
- dcmp
167+
- $(DW_JOB_foo_local_storage)/0
168+
- $(DW_JOB_foo_local_storage)/1
169+
image: dean/red-rock-slushy:v2.0
170+
name: red-rock-launcher
171+
Worker:
172+
template:
173+
spec:
174+
imagePullSecrets:
175+
- name: readonly-red-rock-slushy
176+
containers:
177+
- image: dean/red-rock-slushy:v2.0
178+
name: red-rock-worker
179+
runPolicy:
180+
cleanPodPolicy: Running
181+
suspend: false
182+
slotsPerWorker: 1
183+
sshAuthMountPath: /root/.ssh
184+
pinned: false
185+
retryLimit: 6
186+
storages:
187+
- name: DW_JOB_foo_local_storage
188+
optional: false
189+
- name: DW_PERSISTENT_foo_persistent_storage
190+
optional: true
191+
```
192+
193+
Now any user can select this profile in their Workflow by specifying it in a
194+
`#DW container` directive.
195+
196+
```bash
197+
#DW container profile=mpi-red-rock-slushy [...]
198+
```
199+
200+

0 commit comments

Comments
 (0)