You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -6,17 +6,11 @@ description: A conceptual tour of how Nebari Kubernetes Platform is layered, wha
6
6
7
7
# NKP architecture
8
8
9
-
Nebari Kubernetes Platform (NKP) is the first **Deploy Target** in the Nebari
10
-
ecosystem: the infrastructure that hosts a Nebari platform and runs the
11
-
[Software Packs](/docs/introduction#software-packs) installed on it. This page
12
-
explains how NKP is put together and why, so you can reason about deployments,
13
-
upgrades, and extension points without reading the source.
9
+
Nebari Kubernetes Platform (NKP) is the first **Deploy Target** in the Nebari ecosystem: the infrastructure that hosts a Nebari platform and runs the [Software Packs](/docs/introduction#software-packs) installed on it. This page explains how NKP is put together and why, so you can reason about deployments, upgrades, and extension points without reading the source.
14
10
15
11
## The layers, top to bottom
16
12
17
-
NKP is a stack of layers, each doing one job. Because the layers connect
18
-
through stable, well-defined interfaces, you can change one layer without
19
-
rewriting the others.
13
+
NKP is a stack of layers, each doing one job. Because the layers connect through stable, well-defined interfaces, you can change one layer without rewriting the others.
traffic routing, monitoring, continuous delivery from Git) that power the
39
-
operator and every running pack.
27
+
-**Nebari Operator**: the automation that deploys each Software Pack and connects it to the Foundational software.
28
+
-**Foundational software**: shared services (secure connections, login, traffic routing, monitoring, continuous delivery from Git) that power the operator and every running pack.
40
29
41
30
**Where it all runs**
42
31
43
-
-**Managed Kubernetes cluster**: hosts every container above. EKS, GKE, AKS,
44
-
or K3s for local development.
45
-
-**Cloud or bare-metal provider**: the physical infrastructure underneath.
46
-
AWS, GCP, Azure, dedicated servers (Hetzner, Equinix Metal), or your own
-**Keycloak**: OIDC provider with user federation, MFA, and SSO across
56
-
ArgoCD, Grafana, and Software Packs.
57
-
-**ArgoCD**: GitOps reconciliation (more on this below).
58
-
-**LGTM telemetry stack**: OpenTelemetry Collector, Loki for logs, Grafana
59
-
for visualization, Tempo for traces, Mimir for metrics. Backed by cloud
60
-
object storage for long-term retention.
61
-
62
-
### How the Operator reconciles a pack
63
-
64
-
On every reconcile of a `NebariApplication` (`nebari.dev/v1alpha1`), the
65
-
controller-runtime operator walks four steps in order:
66
-
67
-
1.**Routing**: a cert-manager `Certificate` (if TLS is enabled) and an Envoy
68
-
`HTTPRoute` for the application's domain and paths.
69
-
2.**Authentication**: a Keycloak OAuth2 client scoped to the application's
70
-
allowed groups, users, and public paths; the client secret lands in a
71
-
Kubernetes `Secret`.
72
-
3.**Observability**: a `ServiceMonitor` for metrics and any Grafana
73
-
dashboards declared by the pack (sourced from a URL or a ConfigMap).
74
-
4.**Status**: phase moves through `Pending` → `Provisioning` → `Ready`, and
75
-
the public URL plus per-step conditions are written back to the CRD.
32
+
-**Managed Kubernetes cluster**: hosts every container above. EKS, GKE, AKS, or K3s for local development.
33
+
-**Cloud or bare-metal provider**: the physical infrastructure underneath. AWS, GCP, Azure, dedicated servers (Hetzner, Equinix Metal), or your own machine for development.
34
+
35
+
<details>
36
+
<summary>**What's in the foundational layer**</summary>
37
+
38
+
-**cert-manager**: keeps secure connections working. Automatically requests, renews, and rotates HTTPS certificates for every domain in the cluster.
39
+
-**Envoy Gateway**: handles traffic routing. Inspects incoming requests and forwards each one to the right service, with built-in rate limiting and authentication checks.
40
+
-**Keycloak**: handles login. One sign-on covers every app on the platform (ArgoCD, Grafana, Software Packs), with support for multi-factor authentication and connection to an existing identity provider like Active Directory.
41
+
-**ArgoCD**: keeps the cluster in sync with a Git repository. Whenever the platform team commits a change to Git (a new pack, an updated config), ArgoCD applies it to the cluster automatically.
42
+
-**LGTM telemetry stack**: collects logs, metrics, and traces from every running pack and presents them in shared dashboards.
43
+
44
+
</details>
76
45
77
46
## The `nic` CLI
78
47
79
-
NKP is provisioned by **NIC** (Nebari Infrastructure Core) and its `nic` CLI.
80
-
Understanding the boundary of what `nic` owns is the single most important
81
-
mental model on this page.
48
+
The `nic` CLI (short for **Nebari Infrastructure Core**) is the command-line tool for managing NKP: installing, updating, tearing down, plus inspecting the cloud resources underneath.
82
49
83
50
### What `nic` does
84
51
85
-
-**Provisions the cloud substrate.**`nic` invokes OpenTofu (Terraform)
86
-
modules to create the VPC, the managed Kubernetes cluster, node pools, IAM,
87
-
and durable storage on the chosen provider.
88
-
-**Bootstraps the cluster.** It applies namespaces, RBAC, storage classes, and
89
-
network policies needed before higher layers can run.
90
-
-**Installs ArgoCD.** This is the only Helm chart `nic` installs directly.
91
-
Everything above ArgoCD is reconciled by ArgoCD itself, not by `nic`.
92
-
93
-
That is the boundary. After `nic deploy` finishes, the cluster has Kubernetes,
94
-
the bootstrap resources, and ArgoCD pointed at a Git repository.
52
+
-**Creates the cloud infrastructure.** This includes the network, the managed Kubernetes cluster and its worker machines, the identity and access controls, and the persistent storage. Under the hood, `nic` uses Terraform to do this.
53
+
-**Prepares the cluster.** Sets up the basic Kubernetes structures the platform relies on: organizational groupings (namespaces), permission rules (RBAC), storage templates (storage classes), and network rules (network policies).
54
+
-**Installs ArgoCD.** ArgoCD is the GitOps engine that delivers everything above the cluster (the foundational software, the operator, and the packs). `nic` installs it directly so the rest of the platform can flow in from a Git repository.
95
55
96
56
### What `nic` does not do
97
57
98
-
-**`nic` is not an application manager.** It does not install or upgrade
99
-
Software Packs, the Nebari Operator, or any of the foundational software
100
-
beyond ArgoCD. Those are GitOps concerns (see below).
101
-
-**`nic` does not push to the cluster imperatively.** Once ArgoCD is running,
102
-
changes flow from Git into the cluster, not from a developer's laptop.
58
+
-**`nic` is not an application manager.** It does not install or upgrade Software Packs, the Nebari Operator, or any of the foundational software beyond ArgoCD. Those are all delivered from Git through ArgoCD.
59
+
-**`nic` does not change the cluster directly after install.** Anything that needs to happen inside a running cluster (a new pack, an updated config) goes through Git first, and ArgoCD applies it.
60
+
61
+
This split keeps cluster work (the bottom of the stack) and pack work (the top) from blocking each other. Updating the cluster's underlying machines doesn't affect the running packs, and rolling out a new pack version doesn't require a `nic` deploy.
103
62
104
-
This separation lets infrastructure and applications evolve on independent
105
-
lifecycles: re-rolling a node pool does not touch application state, and
106
-
shipping a new pack version does not require a `nic` run.
(open in a browser to view, or import the HTML into Paper via the write_html MCP tool) */}
65
+

107
66
108
67
## GitOps via ArgoCD
109
68
110
-
Above the cluster, NKP is **declarative and pull-based**. ArgoCD watches a
111
-
Git repository (`nebari-foundational-software`) and continuously reconciles
112
-
the live cluster state to match it.
69
+
Above the cluster, NKP is **declarative and pull-based**. ArgoCD watches a Git repository (`nebari-foundational-software`) and continuously reconciles the live cluster state to match it.
113
70
114
-
ArgoCD uses the **app-of-apps** pattern: a top-level Application resource
115
-
points to a folder of child Applications, each one a piece of foundational
116
-
software with explicit dependencies.
71
+
ArgoCD uses the **app-of-apps** pattern: a top-level Application resource points to a folder of child Applications, each one a piece of foundational software with explicit dependencies.
117
72
118
73
```mermaid
119
74
flowchart LR
@@ -127,23 +82,15 @@ flowchart LR
127
82
128
83
Three properties follow from this:
129
84
130
-
-**Single source of truth.** What is in Git is what is in the cluster. There
131
-
is no "live but undocumented" configuration drift on the application side.
132
-
-**Self-healing.** If an operator deletes a resource by hand, ArgoCD puts it
133
-
back on the next reconciliation tick.
134
-
-**Auditable change.** Every platform change is a Git commit; rollbacks are a
135
-
Git revert.
85
+
-**Single source of truth.** What is in Git is what is in the cluster. There is no "live but undocumented" configuration drift on the application side.
86
+
-**Self-healing.** If an operator deletes a resource by hand, ArgoCD puts it back on the next reconciliation tick.
87
+
-**Auditable change.** Every platform change is a Git commit; rollbacks are a Git revert.
136
88
137
-
Software Packs plug into this model from above. A pack ships its workload
138
-
manifests plus a `NebariApplication` custom resource that declares what the
139
-
pack needs (domain, paths, auth scope, dashboards). Once that custom resource
140
-
lands in the cluster, the Nebari Operator picks it up and runs the four
141
-
reconciliation steps described in the layers section.
89
+
Software Packs plug into this model from above. A pack ships its workload manifests plus a `NebariApplication` custom resource that declares what the pack needs (domain, paths, auth scope, dashboards). Once that custom resource lands in the cluster, the Nebari Operator picks it up and runs the four reconciliation steps described in the layers section.
142
90
143
91
## Terraform state and drift detection
144
92
145
-
`nic` uses a standard **Terraform state file** to track everything it
146
-
provisioned. State is persisted in a remote backend matched to the provider:
93
+
`nic` uses a standard **Terraform state file** to track everything it provisioned. State is persisted in a remote backend matched to the provider:
147
94
148
95
| Provider | Backend | Locking |
149
96
| --- | --- | --- |
@@ -154,13 +101,8 @@ provisioned. State is persisted in a remote backend matched to the provider:
154
101
155
102
Two operational properties matter here:
156
103
157
-
-**Locking prevents concurrent writes.** Two engineers cannot accidentally
158
-
apply conflicting changes at the same time; the second `nic deploy` blocks
159
-
until the first releases the lock.
160
-
-**Drift detection is just `terraform plan`.** Running `nic deploy` against
161
-
an unchanged config produces a diff between the state file and the live
162
-
cloud, surfacing any out-of-band changes (a node group resized in the
163
-
console, an IAM policy edited by hand) as proposed corrections.
104
+
-**Locking prevents concurrent writes.** Two engineers cannot accidentally apply conflicting changes at the same time; the second `nic deploy` blocks until the first releases the lock.
105
+
-**Drift detection is just `terraform plan`.** Running `nic deploy` against an unchanged config produces a diff between the state file and the live cloud, surfacing any out-of-band changes (a node group resized in the console, an IAM policy edited by hand) as proposed corrections.
164
106
165
107
State operations are exposed as first-class CLI verbs:
166
108
@@ -173,16 +115,11 @@ nic state mv <from> <to> # rename a resource address
173
115
174
116
### How this differs from Nebari Classic
175
117
176
-
Classic also used Terraform, but the platform layer above the cluster was
177
-
installed imperatively at deploy time and configured per-environment. NKP
178
-
splits that responsibility: `nic`'s state covers infrastructure only, while
179
-
the platform layer is reconciled by ArgoCD from a Git repository. As a
180
-
result:
118
+
Classic also used Terraform, but the platform layer above the cluster was installed imperatively at deploy time and configured per-environment. NKP splits that responsibility: `nic`'s state covers infrastructure only, while the platform layer is reconciled by ArgoCD from a Git repository. As a result:
181
119
182
120
-**Infrastructure drift** is detectable via `nic deploy` (Terraform plan).
183
121
-**Application drift** is detectable via the ArgoCD UI / CLI (sync status).
184
-
- The two surfaces never overlap, so a Terraform run cannot break an
185
-
application and a pack upgrade cannot rotate a VPC.
122
+
- The two surfaces never overlap, so a Terraform run cannot break an application and a pack upgrade cannot rotate a VPC.
186
123
187
124
## Separation of concerns
188
125
@@ -196,19 +133,13 @@ NKP is built around three roles with non-overlapping lifecycles:
196
133
197
134
The boundary is enforced by the architecture, not by convention:
198
135
199
-
- A pack developer never needs cluster admin or Terraform credentials. They
200
-
publish a pack to the registry; the operator and ArgoCD do the rest.
201
-
- A platform team upgrade (rotating cert-manager, resizing nodes, swapping
202
-
the ingress controller) does not require coordination with pack developers.
203
-
- An end user never sees Kubernetes. They see a Capability on the landing
204
-
page and click into it.
136
+
- A pack developer never needs cluster admin or Terraform credentials. They publish a pack to the registry; the operator and ArgoCD do the rest.
137
+
- A platform team upgrade (rotating cert-manager, resizing nodes, swapping the ingress controller) does not require coordination with pack developers.
138
+
- An end user never sees Kubernetes. They see a Capability on the landing page and click into it.
205
139
206
-
This is the "why" behind the layering: each role can move at its own cadence
207
-
without breaking the others.
140
+
This is the "why" behind the layering: each role can move at its own cadence without breaking the others.
208
141
209
142
## Where to go next
210
143
211
-
-[Get started with NKP](/docs/nkp/get-started): install the platform and
212
-
run your first cluster.
213
-
-[Browse Software Packs](/docs/software-packs/): see what Capabilities are
214
-
available out of the box.
144
+
-[Get started with NKP](/docs/nkp/get-started): install the platform and run your first cluster.
145
+
-[Browse Software Packs](/docs/software-packs/): see what Capabilities are available out of the box.
0 commit comments