Skip to content

Commit f63e6a1

Browse files
maltesandermaltesander
committed
wip - docs
1 parent 68778e1 commit f63e6a1

2 files changed

Lines changed: 284 additions & 0 deletions

File tree

‎docs/modules/demos/pages/argo-cd-git-ops.adoc‎

Lines changed: 283 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,283 @@
1+
= argo-cd-git-ops
2+
:description: Deploy Stackable operators and products with ArgoCD. Sync manifests and secrets from Git.
3+
4+
:k8s-cpu: https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/#cpu
5+
:argo-cd: https://argoproj.github.io/cd/
6+
:argo-cd-application: https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#applications
7+
:argo-cd-applicationset: https://argo-cd.readthedocs.io/en/stable/user-guide/application-set/
8+
:sealed-secrets: https://github.com/bitnami-labs/sealed-secrets
9+
:stackable-demo-repository: https://github.com/stackabletech/demos/
10+
:airflow: https://airflow.apache.org/
11+
12+
This demo shows how to utilize GitOps and Infrastructure as Code (IaC) with Stackable and {argo-cd}[ArgoCD].
13+
All products and manifests are synced and deployed via ArgoCD (except ArgoCD itself, which is bootstrapped via `stackablectl`).
14+
15+
The key points to show are:
16+
* GitOps: How to deploy my changes from a Git repository to my cluster
17+
* Secrets: How to safely deploy credentials, access keys and store them in a Git repository
18+
* Although not included in this Demo, it assumes / tries to imitate a possible
19+
multicluster setup for development, staging and production.
20+
* Interaction with Stackable products (e.g. Airflow and DAGs via gitsync)
21+
22+
Install this demo on an existing Kubernetes cluster:
23+
24+
[source,console]
25+
----
26+
$ `stackablectl` demo install argo-cd-git-ops --namespace argo-cd
27+
----
28+
29+
WARNING: This demo should not be run alongside other demos.
30+
31+
NOTE: ArgoCD will be deployed in the `argo-cd` namespace by `stackablectl`. ArgoCD itself will create other namespaces for the deployed products.
32+
33+
[#system-requirements]
34+
== System requirements
35+
36+
To run this demo, your system needs at least:
37+
38+
* 20 {k8s-cpu}[cpu units] (core/hyperthread)
39+
* 20GiB memory
40+
* 20GiB disk storage
41+
42+
== Overview
43+
44+
This demo consists of multiple parts:
45+
46+
* Bootstrapping via `stackablectl`
47+
** Install a {sealed-secrets}[Sealed Secrets] controller via `stackablectl` to handle sensitive data like credentials or secret keys.
48+
** Install {argo-cd}[Argo CD] via `stackablectl`.
49+
* After the initial bootstrapping, ArgoCD takes over the deployment:
50+
** Install all Stackable operators using an `ApplicationSet`.
51+
** Spin up requirements like Minio and Postgres as `Application`.
52+
** Deploy Stackable Airflow manifests in their respective `Projects`.
53+
** DAGs are synced via Airflow gitsync, not ArgoCD.
54+
** Airflow uses the Kubernetes executor and write its logs to S3 / Minio.
55+
** DAGs can be started manually in the Airflow webserver UI
56+
57+
TODO: arch overview image
58+
59+
== ArgoCD UI
60+
61+
ArgoCD will be the first product that is deployed in this Demo. Once the pods are ready, you can port-forward the argocd-server in order to access the Web UI.
62+
63+
[source,console]
64+
----
65+
kubectl --namespace argo-cd port-forward service/argocd-server 8443:https
66+
----
67+
68+
In your browser, go to `https://localhost:8443` and login with username `admin` and password `adminadmin`.
69+
70+
NOTE: There will be an initial warning from the Browser that this site is insecure due to self-signed certificates. This can be ignored in this case.
71+
72+
TODO: screenshots
73+
74+
== Sealed Secrets
75+
76+
When managing all resources and configs via Git, deploying sensitive properties like certificates or credentials via Git becomes a problem.
77+
78+
There are multiple solutions like Hashicorp or Bitwarden, which heavily depend on the infrastructure already available.
79+
80+
For the sake of this demo, {sealed-secrets}[Bitnami's Sealed Secrets] are utilized.
81+
Sensitive data is encrypted as a `SealedSecret` before commiting to the Git repository, synced via ArgoCD and decrypted by the Sealed Secrets controller into a standard Kubernetes `Secret`.
82+
83+
This way, everything will be stored and managed in Git.
84+
85+
== Stackable operators
86+
87+
The Stackable operators are deployed via ArgoCD using the Stackable Helm charts and an ArgoCD {argo-cd-applicationset}[`ApplicationSet`].
88+
`ApplicationSets` allow templating, which is required to e.g. manage and deploy to multi cluster environments (e.g. development - staging - production),
89+
using different versions and Git sources (repository & branch) as well as the possibility to deploy to different clusters.
90+
91+
NOTE: This demo does not use a multi cluster environment for the sake of simplicity.
92+
93+
[source,console]
94+
----
95+
---
96+
apiVersion: argoproj.io/v1alpha1
97+
kind: ApplicationSet
98+
metadata:
99+
name: stackable-operators
100+
spec:
101+
generators:
102+
- matrix:
103+
generators:
104+
- list:
105+
elements:
106+
- operator: commons
107+
- operator: listener
108+
- operator: secret
109+
- operator: airflow
110+
- operator: druid
111+
- operator: hbase
112+
- operator: hdfs
113+
- operator: hive
114+
- operator: kafka
115+
- operator: nifi
116+
- operator: opa
117+
- operator: spark-k8s
118+
- operator: superset
119+
- operator: trino
120+
- operator: zookeeper
121+
- list:
122+
elements:
123+
- cluster: demo
124+
server: https://kubernetes.default.svc
125+
targetRevision: 25.7.0
126+
###########################################################################################
127+
# The following definitions are not used in this Demo, it is shown for completeness
128+
# for multi cluster setups
129+
###########################################################################################
130+
131+
###########################################################################################
132+
# Development cluster: Checking newest Stackable developments for nightly 0.0.0-dev builds
133+
###########################################################################################
134+
# - cluster: development
135+
# server: https://kubernetes-development.default.svc
136+
# targetRevision: 0.0.0-dev
137+
###########################################################################################
138+
# Staging cluster: Checking compatibility for upgrades from 25.3.0 to 25.7.0
139+
###########################################################################################
140+
# - cluster: staging
141+
# server: https://kubernetes-staging.default.svc
142+
# targetRevision: 25.7.0
143+
###########################################################################################
144+
# Production cluster: Currently running release 25.3.0 and awaiting upgrade to 25.7.0
145+
###########################################################################################
146+
# - cluster: production
147+
# server: https://kubernetes-production.default.svc
148+
# targetRevision: 25.3.0
149+
# [...]
150+
----
151+
152+
The `matrix.generators.list[].elements[]` will create a union of parameters that may be used in the `ApplicationSet` template as follows:
153+
154+
[source,console]
155+
----
156+
# [...]
157+
template:
158+
metadata:
159+
name: "{{ operator }}-operator"
160+
spec:
161+
project: "stackable-operators"
162+
ignoreDifferences:
163+
# mitigating: https://github.com/stackabletech/hdfs-operator/issues/626
164+
- group: "apiextensions.k8s.io"
165+
kind: "CustomResourceDefinition"
166+
jqPathExpressions:
167+
- .spec.names.categories | select(. == [])
168+
- .spec.names.shortNames | select(. == [])
169+
- .spec.versions[].additionalPrinterColumns | select(. == [])
170+
source:
171+
repoURL: "oci.stackable.tech"
172+
targetRevision: "{{ targetRevision }}"
173+
chart: "sdp-charts/{{ operator }}-operator"
174+
helm:
175+
releaseName: "{{ operator }}-operator"
176+
destination:
177+
server: "{{ server }}"
178+
namespace: "stackable-operators"
179+
syncPolicy:
180+
syncOptions:
181+
- CreateNamespace=true
182+
- ServerSideApply=true
183+
- RespectIgnoreDifferences=true
184+
automated:
185+
selfHeal: true
186+
prune: true
187+
----
188+
189+
The templated version for e.g. the parameters `operator=zookeeper`, `server=https://kubernetes.default.svc` and `targetRevision=25.7.0` will result in the following template:
190+
191+
[source,console]
192+
----
193+
# [...]
194+
template:
195+
metadata:
196+
name: "zookeeper-operator"
197+
spec:
198+
project: "stackable-operators"
199+
ignoreDifferences:
200+
- group: "apiextensions.k8s.io"
201+
kind: "CustomResourceDefinition"
202+
jqPathExpressions:
203+
- .spec.names.categories | select(. == [])
204+
- .spec.names.shortNames | select(. == [])
205+
- .spec.versions[].additionalPrinterColumns | select(. == [])
206+
source:
207+
repoURL: "oci.stackable.tech"
208+
targetRevision: "25.7.0"
209+
chart: "sdp-charts/zookeeper-operator"
210+
helm:
211+
releaseName: "zookeeper-operator"
212+
destination:
213+
server: "https://kubernetes.default.svc"
214+
namespace: "stackable-operators"
215+
syncPolicy:
216+
syncOptions:
217+
- CreateNamespace=true
218+
- ServerSideApply=true
219+
- RespectIgnoreDifferences=true
220+
automated:
221+
selfHeal: true
222+
prune: true
223+
----
224+
225+
This allows control over which releases and versions are deployed to which cluster.
226+
227+
Now with ArgoCD deployed, the Sealed Secrets controller and Stackable operators up and running, you can inspect Airflow as the first Stackable product.
228+
229+
== Airflow
230+
231+
The Airflow Web UI is reachable via Nodeport or easier, using a port-forward:
232+
233+
[source,console]
234+
----
235+
kubectl --namespace stackable-airflow port-forward service/airflow-webserver 8080
236+
----
237+
238+
In your browser, go to `https://localhost:8080` and login with username `admin` and password `adminadmin`.
239+
240+
TODO: Screenshots
241+
242+
TODO: Similar to the ArgoCD {argo-cd-application}[`Application`]
243+
244+
=== Starting DAGs
245+
246+
TODO: screenshots?
247+
248+
=== Checking logs
249+
250+
TODO: even important here?
251+
252+
== How to interact with ArgoCD and the Git repository
253+
254+
Since this Demo is hosted in the {stackable-demo-repository}[Stackable Demo repository], where merging etc. requires approval, the recommendation is to fork the {stackable-demo-repository}[Stackable Demo repository].
255+
256+
Once forked, you can install this demo using `stackablectl` parameters to customize the forked repository:
257+
258+
[source,console]
259+
----
260+
stackablectl demo install argo-cd-git-ops --namespace argo-cd --parameters customGitUrl=<my-demo-fork-url> --parameters customGitBranch=<my-custom-branch-with-changes>
261+
----
262+
263+
This way, ArgoCD is instructed to pull the Stackable manifests from the forked repository, where your changes can be properly synced via ArgoCD.
264+
265+
=== Increase Airflow webserver replicas
266+
267+
Assuming your working directory ist the root of the forked demo repository, try to increase the `spec.webservers.roleGroups.<role-group>.replicas` in the folder `demos/argo-cd-git-ops/manifests/airflow/airflow.yaml`.
268+
Once this is pushed / merged, ArgoCD should sync the changes and you should see more webserver pods.
269+
270+
=== Add new Airflow DAGs
271+
272+
Similar to ArgoCD, after adding a new DAG to the folder `demos/argo-cd-git-ops/dags`, Airflow should pick up the new DAG via gitsync.
273+
274+
275+
276+
277+
278+
279+
280+
281+
282+
283+

‎docs/modules/demos/partials/demos.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
* xref:airflow-scheduled-job.adoc[]
2+
* xref:argo-cd-git-ops.adoc[]
23
* xref:data-lakehouse-iceberg-trino-spark.adoc[]
34
* xref:end-to-end-security.adoc[]
45
* xref:hbase-hdfs-load-cycling-data.adoc[]

0 commit comments

Comments
 (0)