Skip to content

Commit c0c44e0

Browse files
soenkeliebaufhennig
soenkeliebau
authored and
fhennig
committed
Replaced breaking change migration notice for hdfs operator with correct method for migration (#351)
* Replaced breaking change migration notice for hdfs operator with correct one. * Fixed include and quotation
1 parent deae917 commit c0c44e0

3 files changed

Lines changed: 190 additions & 35 deletions

File tree

‎modules/ROOT/examples/code/migrate-hdfs-23_1.sh‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
#!/usr/bin/env bash
2+
3+
if [ $# -ne 1 ] ; then
4+
echo "Usage: $0 CLUSTER_NAME"
5+
exit 1
6+
else
7+
HDFS_CLUSTER_NAME=$1
8+
fi
9+
10+
kubectl get pvc -l app.kubernetes.io/name=hdfs -l app.kubernetes.io/instance="$HDFS_CLUSTER_NAME"
11+
for pvc in $(kubectl get pvc -l app.kubernetes.io/name=hdfs -l app.kubernetes.io/instance="$HDFS_CLUSTER_NAME" -l app.kubernetes.io/component=journalnode -o name | sed -e 's#persistentvolumeclaim/##'); do
12+
kubectl apply -f - << EOF
13+
apiVersion: batch/v1
14+
kind: Job
15+
metadata:
16+
name: migrate-journalnode-${pvc}
17+
spec:
18+
template:
19+
spec:
20+
containers:
21+
- name: migrate
22+
image: docker.stackable.tech/stackable/hadoop:3.3.4-stackable23.1.0
23+
command: ["bash", "-c", "ls -la /stackable/data && if [ -d /stackable/data/journal ]; then echo Removing might existing target dir && rm -rf /stackable/data/journalnode && echo Renaming folder && mv /stackable/data/journal /stackable/data/journalnode; else echo Nothing to do; fi"]
24+
volumeMounts:
25+
- name: data
26+
mountPath: /stackable/data
27+
volumes:
28+
- name: data
29+
persistentVolumeClaim:
30+
claimName: ${pvc}
31+
restartPolicy: Never
32+
backoffLimit: 1
33+
EOF
34+
done
35+
for pvc in $(kubectl get pvc -l app.kubernetes.io/name=hdfs -l app.kubernetes.io/instance="$HDFS_CLUSTER_NAME" -l app.kubernetes.io/component=namenode -o name | sed -e 's#persistentvolumeclaim/##'); do
36+
kubectl apply -f - << EOF
37+
apiVersion: batch/v1
38+
kind: Job
39+
metadata:
40+
name: migrate-namenode-${pvc}
41+
spec:
42+
template:
43+
spec:
44+
containers:
45+
- name: migrate
46+
image: docker.stackable.tech/stackable/hadoop:3.3.4-stackable23.1.0
47+
command: ["bash", "-c", "ls -la /stackable/data && if [ -d /stackable/data/name ]; then echo Removing might existing target dir && rm -rf /stackable/data/namenode && echo Renaming folder && mv /stackable/data/name /stackable/data/namenode; else echo Nothing to do; fi"]
48+
volumeMounts:
49+
- name: data
50+
mountPath: /stackable/data
51+
volumes:
52+
- name: data
53+
persistentVolumeClaim:
54+
claimName: ${pvc}
55+
restartPolicy: Never
56+
backoffLimit: 1
57+
EOF
58+
done
59+
for pvc in $(kubectl get pvc -l app.kubernetes.io/name=hdfs -l app.kubernetes.io/instance="$HDFS_CLUSTER_NAME" -l app.kubernetes.io/component=datanode -o name | sed -e 's#persistentvolumeclaim/##'); do
60+
kubectl apply -f - << EOF
61+
apiVersion: batch/v1
62+
kind: Job
63+
metadata:
64+
name: migrate-datanode-${pvc}
65+
spec:
66+
template:
67+
spec:
68+
containers:
69+
- name: migrate
70+
image: docker.stackable.tech/stackable/hadoop:3.3.4-stackable23.1.0
71+
command: ["bash", "-c", "ls -la /stackable/data/data && if [ -d /stackable/data/data/data ]; then echo Removing might existing target dir && rm -rf /stackable/data/data/datanode && echo Renaming folder && mv /stackable/data/data/data /stackable/data/data/datanode; else echo Nothing to do; fi"]
72+
volumeMounts:
73+
- name: data
74+
mountPath: /stackable/data/data
75+
volumes:
76+
- name: data
77+
persistentVolumeClaim:
78+
claimName: ${pvc}
79+
restartPolicy: Never
80+
backoffLimit: 1
81+
EOF
82+
done

‎modules/ROOT/pages/release_notes.adoc‎

Lines changed: 24 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -164,54 +164,43 @@ to
164164
```
165165

166166
==== Stackable Operator for Apache Hadoop
167-
* https://github.com/stackabletech/hdfs-operator/issues/274[Support for multiple storage directories]
167+
* https://github.com/stackabletech/hdfs-operator/issues/290[Enable Log Aggregation for HDFS]
168168

169-
As part of the change mentioned above the naming scheme for the PersistentVolumeClaims written for DataNodes has been changed, so that PVCs written by earlier operator versions are not recognized any more.
170-
Previous PVCs were called `hdfs-datanode-default-0` but now need to have the prefix `data-` added, because otherwise there may be naming collisions due to the ability to specify multiple storage classes.
169+
As part of the change mentioned above we also did some code cleanup that allowed us to remove arbitrary hard-coded values from the operator.
171170

172-
In order to move over existing PVCs a few migration steps are required.
173-
Since it is not possible to rename PVCs you'll need to delete the existing PVCs and recreate them with the correct name and bound to the correct backing PV.
171+
This change affects the directory structure the operator creates inside of the PersistentVolumes used for permanent storage.
174172

175-
Please find an example workflow of how this can be achieved below, this also sets the reclaim policy for the backing PV to `Retain` to avoid the PV being deleted when it becomes unbound, depending on your Kubernetes config this step may not be necessary.
173+
The old folder naming was:
176174

177-
[source,bash]
178-
----
179-
export PVNAME=$(kubectl get pvc hdfs-datanode-default-0 -o yaml | yq '.spec.volumeName')
175+
- DataNode -> `data`
176+
- JournalNode -> `journal`
177+
- NameNode -> `name`
180178

181-
kubectl patch pv ${PVNAME} -p '{"spec":{"persistentVolumeReclaimPolicy": "Retain"}}'
179+
which has now been adopted to match the actual rolename:
182180

183-
kubectl delete pvc hdfs-datanode-default-0
181+
- DataNode -> `datanode`
182+
- JournalNode -> `journalnode`
183+
- NameNode -> `namenode`
184184

185-
kubectl patch pv ${PVNAME} -p '{"spec":{"claimRef": null}}'
186-
----
187185

188-
Afterwards you can recreate the PVC with the new name and bind it to the PV.
186+
Unfortunately, this means that for cluster that where initially rolled out with an older operator version, a one-time migration step becomes necessary to rename these directories.
189187

190-
Please note that you will need to adapt labels, storageClassName and resources to your specific configuration.
191-
Ideally export the pre-existing PVC with kubectl and change the name.
188+
You can either do this manually by attaching the PVs to a pod and performing the rename (cluster needs to be stopped for this) or use the script provided below.
192189

193-
[source,yaml]
190+
WARNING: Please be aware that if this script runs after the cluster was already restarted with the newer operator version it will delete any data that was written to the empty post-upgrade HDFS that was stood up by the new operator.
191+
192+
[source,bash]
194193
----
195-
apiVersion: v1
196-
kind: PersistentVolumeClaim
197-
metadata:
198-
labels:
199-
app.kubernetes.io/component: datanode
200-
app.kubernetes.io/instance: hdfs
201-
app.kubernetes.io/name: hdfs
202-
app.kubernetes.io/role-group: default
203-
name: data-hdfs-datanode-default-0
204-
spec:
205-
accessModes:
206-
- ReadWriteOnce
207-
resources:
208-
requests:
209-
storage: 1Gi
210-
storageClassName: standard
211-
volumeMode: Filesystem
212-
volumeName: <insert PV Name here>
194+
include::example$code/migrate-hdfs-23_1.sh[]
213195
----
214196

197+
The migration process for this now becomes:
198+
199+
* Stop HDFS cluster by either removing the HdfsCluster definition object or scaling all roles to 0 replicas
200+
* Uninstall Stackable Operator for Apache Hadoop
201+
* Run migration script
202+
* Install newer version of Stackable Operator for Apache Hadoop
203+
215204
==== Stackable Operator for Apache Hive
216205
* https://github.com/stackabletech/hive-operator/pull/292[Moved database specification from role/role-group level to top-level clusterConfig]
217206
* https://github.com/stackabletech/hive-operator/pull/292[Moved s3, serviceType and hdfs discovery to top-level clusterConfig]

‎modules/contributor/pages/adr/drafts/ADRx-maintenance_windows.adoc‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
= Definition of Maintenance Windows for the Managed Applications
2+
Sönke Liebau <soenke.liebau@stackable.tech>
3+
v0.1, 2023-01-30
4+
:status: draft
5+
6+
* Status: {status}
7+
* Deciders: [list everyone involved in the decision] <!-- optional -->
8+
* Date: [YYYY-MM-DD when the decision was last updated] <!-- optional -->
9+
10+
Technical Story: [description | ticket/issue URL] <!-- optional -->
11+
12+
== Context and Problem Statement
13+
14+
When operating productive environments with the tools of the SDP, it can often be desirable to avoid restarts or changes unless they have been planned and communicated.
15+
16+
There are a couple of things that could trigger a change that in turn trigger a restart of the applications in the SDP:
17+
18+
- User changes to the definitions (CRDs)
19+
- Operator upgrade that causes different objects to be written for the same definition
20+
- Changes to dependent objects
21+
- ...
22+
23+
In order to enable users to better control when these changes are allowed to affect deployed applications we should add the ability to define maintenance windows
24+
25+
== Decision Drivers <!-- optional -->
26+
27+
* [driver 1, e.g., a force, facing concern, …]
28+
* [driver 2, e.g., a force, facing concern, …]
29+
* … <!-- numbers of drivers can vary -->
30+
31+
== Considered Options
32+
33+
* Implement a top level struct that can in
34+
* [option 2]
35+
* [option 3]
36+
* … <!-- numbers of options can vary -->
37+
38+
== Decision Outcome
39+
40+
Chosen option: "[option 1]", because [justification. e.g., only option, which meets k.o. criterion decision driver | which resolves force force | … | comes out best (see below)].
41+
42+
=== Positive Consequences <!-- optional -->
43+
44+
* [e.g., improvement of quality attribute satisfaction, follow-up decisions required, …]
45+
* …
46+
47+
=== Negative Consequences <!-- optional -->
48+
49+
* [e.g., compromising quality attribute, follow-up decisions required, …]
50+
* …
51+
52+
== Pros and Cons of the Options <!-- optional -->
53+
54+
=== [option 1]
55+
56+
[example | description | pointer to more information | …] <!-- optional -->
57+
58+
* Good, because [argument a]
59+
* Good, because [argument b]
60+
* Bad, because [argument c]
61+
* … <!-- numbers of pros and cons can vary -->
62+
63+
=== [option 2]
64+
65+
[example | description | pointer to more information | …] <!-- optional -->
66+
67+
* Good, because [argument a]
68+
* Good, because [argument b]
69+
* Bad, because [argument c]
70+
* … <!-- numbers of pros and cons can vary -->
71+
72+
=== [option 3]
73+
74+
[example | description | pointer to more information | …] <!-- optional -->
75+
76+
* Good, because [argument a]
77+
* Good, because [argument b]
78+
* Bad, because [argument c]
79+
* … <!-- numbers of pros and cons can vary -->
80+
81+
== Links <!-- optional -->
82+
83+
* [Link type] [Link to ADR] <!-- example: Refined by [ADR-0005](0005-example.md) -->
84+
* … <!-- numbers of links can vary -->

0 commit comments

Comments
 (0)