Skip to content

Commit 6c56701

Browse files
matthew-richersonmatthew-richerson
authored
Merge pull request #103 from NearNodeFlash/release-v0.0.4
Release v0.0.4
2 parents c2b40f7 + 2ab47e6 commit 6c56701

6 files changed

Lines changed: 733 additions & 76 deletions

File tree

‎docs/guides/external-mgs/readme.md‎

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
---
2+
authors: Matt Richerson <matthew.richerson@hpe.com>
3+
categories: provisioning
4+
---
5+
6+
# Lustre External MGT
7+
8+
## Background
9+
10+
Lustre has a limitation where only a single MGT can be mounted on a node at a time. In some situations it may be desirable to share an MGT between multiple Lustre file systems to increase the number of Lustre file systems that can be created and to decrease scheduling complexity. This guide provides instructions on how to configure NNF to share MGTs. There are three methods that can be used:
11+
12+
1. Use a Lustre MGT from outside the NNF cluster
13+
2. Create a persistent Lustre file system through DWS and use the MGT it provides
14+
3. Create a pool of standalone persistent Lustre MGTs, and have the NNF software select one of them
15+
16+
These three methods are not mutually exclusive on the system as a whole. Individual file systems can use any of options 1-3 or create their own MGT.
17+
18+
## Configuration with an External MGT
19+
20+
An existing MGT external to the NNF cluster can be used to manage the Lustre file systems on the NNF nodes. An advantage to this configuration is that the MGT can be highly available through multiple MGSs. A disadvantage is that there is only a single MGT. An MGT shared between more than a handful of Lustre file systems is not a common use case, so the Lustre code may prove less stable.
21+
22+
The following yaml provides an example of what the `NnfStorageProfile` should contain to use an MGT on an external server.
23+
24+
```yaml
25+
apiVersion: nnf.cray.hpe.com/v1alpha1
26+
kind: NnfStorageProfile
27+
metadata:
28+
name: external-mgt
29+
namespace: nnf-system
30+
data:
31+
[...]
32+
lustreStorage:
33+
externalMgs: 1.2.3.4@eth0
34+
combinedMgtMdt: false
35+
standaloneMgtPoolName: ""
36+
[...]
37+
```
38+
39+
## Configuration with Persistent Lustre
40+
41+
The MGT from a persistent Lustre file system hosted on the NNF nodes can also be used as the MGT for other NNF Lustre file systems. This configuration has the advantage of not relying on any hardware outside of the cluster. However, there is no high availability, and a single MGT is still shared between all Lustre file systems created on the cluster.
42+
43+
To configure a persistent Lustre file system that can share its MGT, a `NnfStorageProfile` should be used that does not specify `externalMgs`. The MGT can either share a volume with the MDT or not (`combinedMgtMdt`).
44+
45+
```yaml
46+
apiVersion: nnf.cray.hpe.com/v1alpha1
47+
kind: NnfStorageProfile
48+
metadata:
49+
name: persistent-lustre-shared-mgt
50+
namespace: nnf-system
51+
data:
52+
[...]
53+
lustreStorage:
54+
externalMgs: ""
55+
combinedMgtMdt: false
56+
standaloneMgtPoolName: ""
57+
[...]
58+
```
59+
60+
The persistent storage is created with the following DW directive:
61+
62+
```bash
63+
#DW create_persistent name=shared-lustre capacity=100GiB type=lustre profile=persistent-lustre-shared-mgt
64+
```
65+
66+
After the persistent Lustre file system is created, an admin can discover the MGS address by looking at the `NnfStorage` resource with the same name as the persistent storage that was created (`shared-lustre` in the above example).
67+
68+
```yaml
69+
apiVersion: nnf.cray.hpe.com/v1alpha1
70+
kind: NnfStorage
71+
metadata:
72+
name: shared-lustre
73+
namespace: default
74+
[...]
75+
status:
76+
mgsNode: 5.6.7.8@eth1
77+
[...]
78+
```
79+
80+
A separate `NnfStorageProfile` can be created that specifies the MGS address.
81+
82+
```yaml
83+
apiVersion: nnf.cray.hpe.com/v1alpha1
84+
kind: NnfStorageProfile
85+
metadata:
86+
name: internal-mgt
87+
namespace: nnf-system
88+
data:
89+
[...]
90+
lustreStorage:
91+
externalMgs: 5.6.7.8@eth1
92+
combinedMgtMdt: false
93+
standaloneMgtPoolName: ""
94+
[...]
95+
```
96+
97+
With this configuration, an admin must determine that no file systems are using the shared MGT before destroying the persistent Lustre instance.
98+
99+
## Configuration with an Internal MGT Pool
100+
101+
Another method NNF supports is to create a number of persistent Lustre MGTs on NNF nodes. These MGTs are not part of a full file system, but are instead added to a pool of MGTs available for other Lustre file systems to use. Lustre file systems that are created will choose one of the MGTs at random to use and add a reference to make sure it isn't destroyed. This configuration has the advantage of spreading the Lustre management load across multiple servers. The disadvantage of this configuration is that it does not provide high availability.
102+
103+
To configure the system this way, the first step is to make a pool of Lustre MGTs. This is done by creating a persistent instance from a storage profile that specifies the `standaloneMgtPoolName` option. This option tells NNF software to only create an MGT, and to add it to a named pool. The following `NnfStorageProfile` provides an example where the MGT is added to the `example-pool` pool:
104+
105+
```yaml
106+
apiVersion: nnf.cray.hpe.com/v1alpha1
107+
kind: NnfStorageProfile
108+
metadata:
109+
name: mgt-pool-member
110+
namespace: nnf-system
111+
data:
112+
[...]
113+
lustreStorage:
114+
externalMgs: ""
115+
combinedMgtMdt: false
116+
standaloneMgtPoolName: "example-pool"
117+
[...]
118+
```
119+
120+
A persistent storage MGTs can be created with the following DW directive:
121+
122+
```bash
123+
#DW create_persistent name=mgt-pool-member-1 capacity=1GiB type=lustre profile=mgt-pool-member
124+
```
125+
126+
Multiple persistent instances with different names can be created using the `mgt-pool-member` profile to add more than one MGT to the pool.
127+
128+
To create a Lustre file system that uses one of the MGTs from the pool, an `NnfStorageProfile` should be created that uses the special notation `pool:[pool-name]` in the `externalMgs` field.
129+
130+
```yaml
131+
apiVersion: nnf.cray.hpe.com/v1alpha1
132+
kind: NnfStorageProfile
133+
metadata:
134+
name: mgt-pool-consumer
135+
namespace: nnf-system
136+
data:
137+
[...]
138+
lustreStorage:
139+
externalMgs: "pool:example-pool"
140+
combinedMgtMdt: false
141+
standaloneMgtPoolName: ""
142+
[...]
143+
```
144+
145+
The following provides an example DW directive that uses an MGT from the MGT pool:
146+
147+
```bash
148+
#DW jobdw name=example-lustre capacity=100GiB type=lustre profile=mgt-pool-consumer
149+
```
150+
151+
MGT pools are named, so there can be separate pools with collections of different MGTs in them. A storage profile targeting each pool would be needed.

‎docs/guides/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
* [Storage Profiles](storage-profiles/readme.md)
1414
* [Data Movement Configuration](data-movement/readme.md)
15+
* [Lustre External MGT](external-mgs/readme.md)
1516

1617
## NNF User Containers
1718

0 commit comments

Comments
 (0)