|
| 1 | +<!-- |
| 2 | + Licensed to the Apache Software Foundation (ASF) under one |
| 3 | + or more contributor license agreements. See the NOTICE file |
| 4 | + distributed with this work for additional information |
| 5 | + regarding copyright ownership. The ASF licenses this file |
| 6 | + to you under the Apache License, Version 2.0 (the |
| 7 | + "License"); you may not use this file except in compliance |
| 8 | + with the License. You may obtain a copy of the License at |
| 9 | +
|
| 10 | + http://www.apache.org/licenses/LICENSE-2.0 |
| 11 | +
|
| 12 | + Unless required by applicable law or agreed to in writing, |
| 13 | + software distributed under the License is distributed on an |
| 14 | + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| 15 | + KIND, either express or implied. See the License for the |
| 16 | + specific language governing permissions and limitations |
| 17 | + under the License. |
| 18 | +--> |
| 19 | + |
| 20 | +# Apache CloudStack Kubernetes Provider Security Threat Model — delta (draft) |
| 21 | + |
| 22 | +> **Delta document.** Inherits §3, §4 B1, and §7 from |
| 23 | +> `cloudstack-threat-model-draft.md`. Read the main model first. |
| 24 | +
|
| 25 | +## §1 Header |
| 26 | + |
| 27 | +- **Project:** Apache CloudStack Kubernetes Provider |
| 28 | + (`apache/cloudstack-kubernetes-provider`) — Kubernetes Cloud |
| 29 | + Controller Manager (CCM) for CloudStack-managed clusters. |
| 30 | +- **Commit:** `4740dbc` (HEAD of `main` at draft time). |
| 31 | +- **Date:** 2026-05-29. |
| 32 | +- **Authors:** ASF Security team draft. |
| 33 | +- **Status:** Draft delta over `cloudstack-threat-model-draft.md`. |
| 34 | +- **Version binding:** as of the commit above. |
| 35 | +- **Reporting:** as in the main model. |
| 36 | +- **Provenance legend:** as in the main model. |
| 37 | +- **Draft confidence:** 11 documented / 0 maintainer / 12 inferred. |
| 38 | + |
| 39 | +**About the project.** A Kubernetes Cloud Controller Manager (CCM) |
| 40 | +written in Go that lets a Kubernetes cluster running on CloudStack |
| 41 | +discover its node instances, sync labels, and provision load |
| 42 | +balancers via the CloudStack API *(documented: `README.md`)*. Replaces |
| 43 | +the old in-tree `kubernetes/kubernetes` CloudStack provider that was |
| 44 | +removed *(documented: `README.md` — references to |
| 45 | +`kubernetes/enhancements` issues #672 and #88)*. Deployed automatically |
| 46 | +when CloudStack 4.16+ creates a Kubernetes cluster; can also be |
| 47 | +deployed manually. Runs as a Pod in the `kube-system` namespace. |
| 48 | + |
| 49 | +**Crucially**, *(documented: `README.md` "Deployment")*: a separate |
| 50 | +service user **`kubeadmin`** is created in the same CloudStack account |
| 51 | +as the cluster owner, and the CCM uses **that user's API keys** to |
| 52 | +talk to CloudStack. The CCM holds long-lived CloudStack credentials in |
| 53 | +a Kubernetes Secret. |
| 54 | + |
| 55 | +## §2 Scope and intended use |
| 56 | + |
| 57 | +**Primary intended use.** Run as the Kubernetes Cloud Controller |
| 58 | +Manager for a Kubernetes cluster whose nodes are CloudStack-managed |
| 59 | +VMs. Discovers node identity, propagates zone / region labels, |
| 60 | +and provisions CloudStack load balancers in response to `Service |
| 61 | +type=LoadBalancer` declarations *(documented: `cloudstack.go`, |
| 62 | +`cloudstack_loadbalancer.go`, `cloudstack_instances.go`)*. |
| 63 | + |
| 64 | +**Deployment shape.** A long-running Pod inside `kube-system`. Has |
| 65 | +network access to (a) the Kubernetes API server (in-cluster |
| 66 | +ServiceAccount auth), and (b) the CloudStack management server |
| 67 | +(`api-url` from the mounted `cloud-config` Secret). |
| 68 | + |
| 69 | +**Caller expectations.** The Kubernetes cluster operator is trusted |
| 70 | +to: |
| 71 | + |
| 72 | +- create the `cloudstack-secret` containing `cloud-config` with |
| 73 | + `api-url`, `api-key`, `secret-key`, optional `project-id`, `zone`, |
| 74 | + `region`, and **`ssl-no-verify`** *(documented: `README.md`)*, |
| 75 | +- not regenerate the `kubeadmin` user's API keys after deployment |
| 76 | + (the CCM relies on a stable identity) *(documented: `README.md`)*, |
| 77 | +- restrict access to the `cloudstack-secret` per Kubernetes RBAC. |
| 78 | + |
| 79 | +**Component-family table.** |
| 80 | + |
| 81 | +| Family | Representative entry | Touches outside the process? | In this delta? | |
| 82 | +| --- | --- | --- | --- | |
| 83 | +| Provider config + CloudStack client (`cloudstack.go` `CSConfig`, `newCSCloud`) | reads `cloud-config` Secret, builds `cloudstack-go` client | **yes — network + creds** | yes | |
| 84 | +| Instance discovery (`cloudstack_instances.go`) | maps node VM IDs to CloudStack VMs | inherited | yes | |
| 85 | +| Load-balancer reconciler (`cloudstack_loadbalancer.go`) | provisions CloudStack LB rules in response to K8s Service changes | inherited | yes | |
| 86 | +| Protocol helpers (`protocol.go`) | TCP / UDP / TCP-Proxy LoadBalancer support *(documented: `README.md`)* | n/a | yes | |
| 87 | +| `cmd/cloudstack-ccm` | CCM entry point | binds in-cluster ServiceAccount | yes | |
| 88 | +| `deployment.yaml`, `nginx-ingress-controller-patch.yml`, Dockerfile | packaging | n/a at runtime | yes — these define the *deployment posture* | |
| 89 | +| Tests `*_test.go` | unit / integration tests | n/a | **out of model** *(§3)* | |
| 90 | +| `get_kubernetes_deps.sh` | dev script | n/a | **out of model** *(§3)* | |
| 91 | + |
| 92 | +## §3 Out of scope (explicit non-goals) |
| 93 | + |
| 94 | +The main model's §3 applies. **Additional** out-of-scope items |
| 95 | +specific to the K8s provider: |
| 96 | + |
| 97 | +1. **Kubernetes Secret confidentiality.** The CCM expects `cloud-config` |
| 98 | + to be in a Kubernetes Secret. Whether that Secret is encrypted at |
| 99 | + rest (etcd encryption, KMS provider), bound to a specific Pod, or |
| 100 | + exposed via in-cluster reads is the Kubernetes cluster's own |
| 101 | + responsibility. *(inferred — Q1)* |
| 102 | +2. **Server-side correctness of CloudStack responses.** Same as |
| 103 | + sibling deltas. |
| 104 | +3. **TLS verification when `ssl-no-verify = true`.** When the cluster |
| 105 | + operator sets `ssl-no-verify = true` in `cloud-config`, the CCM |
| 106 | + passes `verifyssl=false` to `cloudstack-go` (via the equivalent |
| 107 | + constructor). That is operator choice. |
| 108 | +4. **In-cluster RBAC for the CCM's ServiceAccount.** What Kubernetes |
| 109 | + API permissions the CCM holds via its ServiceAccount and Role |
| 110 | + bindings is the cluster operator's job. |
| 111 | +5. **Kubernetes itself.** Bugs in `kube-apiserver`, `kubelet`, etcd, |
| 112 | + the CRI runtime are upstream. |
| 113 | +6. **The `kubeadmin` user creation flow on the CloudStack side.** The |
| 114 | + `kubeadmin` user is created by CloudStack's `cloudstack-management` |
| 115 | + when the cluster is provisioned; that flow is in the main model. |
| 116 | + This delta covers only the CCM's *consumption* of those credentials. |
| 117 | +7. **The four sibling repos.** |
| 118 | + |
| 119 | +## §4 Trust boundaries and data flow |
| 120 | + |
| 121 | +The CCM stitches together two main-model trust transitions: |
| 122 | + |
| 123 | +- **K1: K8s controller-manager → Kubernetes API server.** In-cluster |
| 124 | + ServiceAccount token, mounted into the Pod. Authn / authz are |
| 125 | + Kubernetes-side. |
| 126 | +- **K2: CCM → CloudStack management server.** Main-model B1 — HMAC- |
| 127 | + SHA1 signed JSON API calls. Credentials come from the |
| 128 | + `cloudstack-secret`'s `cloud-config`. |
| 129 | + |
| 130 | +The boundary is **the Kubernetes Pod sandbox** plus the mounted |
| 131 | +Secret. Bytes inside the Pod that come from either the K8s API server |
| 132 | +or the CloudStack management server are trusted control-plane content. |
| 133 | + |
| 134 | +**`ssl-no-verify` behaviour** *(documented: `cloudstack.go` `CSConfig` |
| 135 | +gcfg tag `ssl-no-verify`)*: when `true`, TLS verification of the |
| 136 | +CloudStack management-server cert is disabled. |
| 137 | + |
| 138 | +## §5 Assumptions about the environment |
| 139 | + |
| 140 | +- **Host**: Kubernetes 1.16+ (the in-tree provider was removed in |
| 141 | + 1.15) *(documented: `README.md`)*. |
| 142 | +- **Container**: `apache/cloudstack-kubernetes-provider` published on |
| 143 | + Docker Hub *(documented: `README.md`)*. |
| 144 | +- **Filesystem**: reads `cloud-config` from a mounted Secret; |
| 145 | + otherwise no host filesystem writes *(inferred — Q2)*. |
| 146 | +- **Network**: outbound HTTPS to `api-url` (CloudStack) and in-cluster |
| 147 | + HTTPS to `kube-apiserver`. |
| 148 | +- **Auth**: stdlib TLS, opt-out via `ssl-no-verify=true`; in-cluster |
| 149 | + K8s auth via projected ServiceAccount token. |
| 150 | +- **What the CCM does not do**: no host-network listener, no privileged |
| 151 | + Pod requirement *(inferred — Q3)*, no write to the host filesystem. |
| 152 | + |
| 153 | +## §5a Build-time and configuration variants |
| 154 | + |
| 155 | +| Knob | Default | Stance | Effect | |
| 156 | +| --- | --- | --- | --- | |
| 157 | +| `api-url` | none | operator config | endpoint | |
| 158 | +| `api-key`, `secret-key` | none — provisioned automatically as `kubeadmin` | operator must not regenerate after deployment *(documented: `README.md`)* | identity | |
| 159 | +| `ssl-no-verify` | `false` *(inferred — Q4)* | dev / self-signed-CA | flips TLS verification off | |
| 160 | +| `project-id`, `zone`, `region` | unset | optional scoping | constrains LB / instance discovery to a project | |
| 161 | +| LoadBalancer protocols | TCP, UDP, TCP-Proxy *(documented: `README.md`)* | supported set | not security-relevant | |
| 162 | + |
| 163 | +## §6 Assumptions about inputs |
| 164 | + |
| 165 | +| Entry point | Parameter | Attacker-controllable in the model? | Caller must enforce | |
| 166 | +| --- | --- | --- | --- | |
| 167 | +| `cloud-config` Secret | `api-url`, `api-key`, `secret-key`, `ssl-no-verify`, etc. | **no** — Kubernetes cluster operator config | restrict Secret read RBAC; consider etcd-at-rest encryption | |
| 168 | +| Kubernetes Service / Node / Endpoint events | watch payloads | **trusted from kube-apiserver** | bytes are control-plane | |
| 169 | +| CloudStack JSON responses | typed-decoded | trusted (B1) | bytes are control-plane | |
| 170 | + |
| 171 | +## §7 Adversary model |
| 172 | + |
| 173 | +Main-model §7 applies. **Adjustments specific to the K8s provider**: |
| 174 | + |
| 175 | +- "Unauthenticated network peer reaching `:8080`" is upstream. |
| 176 | +- An additional adversary worth naming: **any Kubernetes principal |
| 177 | + with `get`/`list` on Secrets in `kube-system`**. Such a principal |
| 178 | + recovers the CloudStack `kubeadmin` user's API key and secret in |
| 179 | + plaintext and can drive the CloudStack API as `kubeadmin` (with |
| 180 | + whatever CloudStack-side role that user holds). This is in scope: |
| 181 | + the CCM cannot prevent in-cluster Secret read, but the documented |
| 182 | + *(README)* deployment recommendation that `kubeadmin` keys must not |
| 183 | + be regenerated means a leaked `cloud-config` cannot be invalidated |
| 184 | + by rotation without breaking the CCM. *(inferred — Q5 — high- |
| 185 | + priority.)* |
| 186 | +- **A passive observer on the in-cluster network** between the CCM |
| 187 | + Pod and the CloudStack management server when `ssl-no-verify=true` |
| 188 | + is the same shape as in the Terraform-provider delta: TLS verify is |
| 189 | + off, MitM is possible undetected. |
| 190 | + |
| 191 | +## §8 Security properties the CCM provides |
| 192 | + |
| 193 | +### K1 — HMAC-SHA1 signature via `cloudstack-go` |
| 194 | + |
| 195 | +- As main-model §8 P1. |
| 196 | + |
| 197 | +### K2 — In-cluster ServiceAccount-bound CCM identity |
| 198 | + |
| 199 | +- **Property.** The CCM authenticates to the Kubernetes API server as |
| 200 | + its own ServiceAccount; no shared secret is sent to kube-apiserver. |
| 201 | +- **Conditions.** Pod is deployed per `deployment.yaml` with a |
| 202 | + ServiceAccount. |
| 203 | +- **Violation symptom.** CCM speaks to kube-apiserver as a different |
| 204 | + identity. |
| 205 | +- **Severity.** Security-critical, `VALID`. |
| 206 | + |
| 207 | +### K3 — TLS verification toggle (`ssl-no-verify`) |
| 208 | + |
| 209 | +- As Go-SDK delta §8 S2 / CloudMonkey delta §8 C2. |
| 210 | + |
| 211 | +## §9 Security properties the CCM does *not* provide |
| 212 | + |
| 213 | +- **No protection of the `cloud-config` Secret.** A K8s principal with |
| 214 | + Secret read in `kube-system` recovers CloudStack credentials. |
| 215 | +- **No key rotation.** The README *requires* operators **not** to |
| 216 | + rotate `kubeadmin` API keys after deployment — meaning a compromise |
| 217 | + cannot be remediated by rotation without breaking the CCM. *(See |
| 218 | + §14 Q6 for the proposed-rotation question.)* |
| 219 | +- **No defence when `ssl-no-verify = true`.** |
| 220 | +- **No least-privilege constraint** on the `kubeadmin` CloudStack user's |
| 221 | + scope beyond what CloudStack's RBAC + project / zone scoping gives. |
| 222 | + *(inferred — Q7)* |
| 223 | +- **No defence against an attacker with read access to the CCM Pod's |
| 224 | + in-cluster network namespace** (e.g. a sidecar in the same Pod). |
| 225 | + *(inferred — Q8)* |
| 226 | + |
| 227 | +### False-friend properties |
| 228 | + |
| 229 | +- **`ssl-no-verify` is not "test mode."** Same wording as the |
| 230 | + Terraform delta. |
| 231 | +- **A Kubernetes Secret is not a hardened secret store** — it is |
| 232 | + base64-encoded by default in etcd, encrypted only when etcd |
| 233 | + encryption is enabled. |
| 234 | +- **HMAC-SHA1** — see main model §11a. |
| 235 | + |
| 236 | +## §10 Downstream responsibilities |
| 237 | + |
| 238 | +The Kubernetes cluster operator MUST: |
| 239 | + |
| 240 | +1. Enable etcd at-rest encryption (or KMS-backed Secret encryption) |
| 241 | + for the `cloudstack-secret`. |
| 242 | +2. Restrict Kubernetes RBAC so that only the CCM ServiceAccount can |
| 243 | + read `cloudstack-secret`. |
| 244 | +3. Set `ssl-no-verify = false` (the recommended posture) unless the |
| 245 | + CloudStack management server is unreachable over TLS in the |
| 246 | + cluster's network. |
| 247 | +4. Scope the CloudStack-side `kubeadmin` user to the smallest |
| 248 | + CloudStack role that can list VMs and create/update load |
| 249 | + balancers — not a root admin or domain admin. |
| 250 | +5. Treat the CCM Pod as a credential-bearing workload: no co-tenant |
| 251 | + sidecars, no shared PID namespace with untrusted workloads. |
| 252 | +6. Match CCM container image to the CloudStack API contract (the |
| 253 | + image embeds a specific `cloudstack-go` version). |
| 254 | +7. Monitor CloudStack audit logs for `kubeadmin`-account anomalies. |
| 255 | + |
| 256 | +## §11 Known misuse patterns |
| 257 | + |
| 258 | +- Storing `cloud-config` in a Kubernetes Secret without etcd |
| 259 | + encryption. |
| 260 | +- Granting `secret get` on `kube-system` to non-admin Kubernetes |
| 261 | + principals. |
| 262 | +- Running the CCM with a `kubeadmin` user that is a root admin or |
| 263 | + domain admin of the CloudStack account (over-privileged). |
| 264 | +- Setting `ssl-no-verify = true` in production. |
| 265 | +- Sharing the `cloud-config` across clusters of different trust |
| 266 | + levels — a compromise of one cluster's Secret compromises the |
| 267 | + others. |
| 268 | +- Regenerating the `kubeadmin` API key — breaks the CCM until the |
| 269 | + `cloud-config` Secret is updated *(documented: `README.md`)*. |
| 270 | + |
| 271 | +## §11a Known non-findings (recurring false positives) |
| 272 | + |
| 273 | +- **"Secret contains plaintext `api-key` / `secret-key`."** That is |
| 274 | + the documented deployment shape. → `BY-DESIGN: property-disclaimed`. |
| 275 | +- **"Pod has access to a long-lived credential."** Same. → `BY-DESIGN: |
| 276 | + property-disclaimed`. |
| 277 | +- **"HMAC-SHA1 — SHA1 is deprecated."** → `KNOWN-NON-FINDING` per |
| 278 | + main model. |
| 279 | +- **"`InsecureSkipVerify` is reachable from `ssl-no-verify`."** |
| 280 | + Operator choice. → `OUT-OF-MODEL: trusted-input`. |
| 281 | +- **"`kubeadmin` user has broad CloudStack privileges."** Depends on |
| 282 | + operator's CloudStack RBAC posture per §10 item 4. → `OUT-OF-MODEL: |
| 283 | + trusted-input` against the operator's CloudStack config. |
| 284 | +- **"Tests in `*_test.go` have weak input handling."** Out of model. |
| 285 | + → `OUT-OF-MODEL: unsupported-component`. |
| 286 | + |
| 287 | +## §12 Conditions that would change this delta |
| 288 | + |
| 289 | +- A move from static `cloud-config` Secret to a secret-manager |
| 290 | + pattern (External Secrets Operator, CSI Secret Driver, projected |
| 291 | + short-lived token from the CloudStack side). |
| 292 | +- Support for short-lived / rotating credentials at the CloudStack |
| 293 | + side (would change the §9 "no key rotation" bullet). |
| 294 | +- Addition of a CCM-side TLS-verify default that overrides |
| 295 | + `ssl-no-verify=false` to `true`. |
| 296 | +- Change in signing algorithm at the main-model layer. |
| 297 | + |
| 298 | +## §13 Triage dispositions |
| 299 | + |
| 300 | +Use the same table as the main model. |
| 301 | + |
| 302 | +## §14 Open questions for the maintainers |
| 303 | + |
| 304 | +**Q1.** Out-of-scope: Kubernetes-side Secret confidentiality (etcd |
| 305 | +encryption, KMS, RBAC). Confirm. *(maps to §3, §9)* |
| 306 | + |
| 307 | +**Q2.** Confirm the CCM has no host-filesystem writes and no |
| 308 | +ephemeral credential cache that survives Pod restart. |
| 309 | + |
| 310 | +**Q3.** Does the upstream `deployment.yaml` require any host |
| 311 | +privileges (host network, host PID, privileged container)? Proposed: |
| 312 | +**no**. Confirm. |
| 313 | + |
| 314 | +**Q4.** `ssl-no-verify` default — proposed: **`false`**. Confirm. |
| 315 | +*(maps to §5a, §10)* |
| 316 | + |
| 317 | +**Q5.** What is the recommended Kubernetes Secret protection posture |
| 318 | +for `cloudstack-secret`? Proposed: etcd-at-rest encryption + RBAC |
| 319 | +restricting `get`/`list` on Secrets in `kube-system` to cluster |
| 320 | +admins + the CCM ServiceAccount only. *(maps to §3, §10)* |
| 321 | + |
| 322 | +**Q6.** **Highest-leverage question in this delta.** The README says |
| 323 | +*"It is imperative that this user is not altered or have its keys |
| 324 | +regenerated."* — meaning credential rotation is not supported. |
| 325 | + |
| 326 | +- Is rotation actually impossible, or is it documented-not-supported |
| 327 | + because the operator-side workflow is non-trivial? |
| 328 | +- Is there a path to rotate the `kubeadmin` API key with controlled |
| 329 | + CCM downtime? |
| 330 | +- Should the threat model state explicitly that the CCM's |
| 331 | + CloudStack credential is *non-rotatable* and treat any leak as |
| 332 | + requiring redeployment of the cluster's CloudStack account? |
| 333 | + *(maps to §9, §10, §11)* |
| 334 | + |
| 335 | +**Q7.** What CloudStack-side RBAC scope is `kubeadmin` *expected* to |
| 336 | +have? Proposed: the smallest set of API commands that covers `listVMs` |
| 337 | ++ LoadBalancer rule CRUD. Is there a published recommended role? |
| 338 | +*(maps to §9, §10)* |
| 339 | + |
| 340 | +**Q8.** What is the CCM's posture against a malicious sidecar in the |
| 341 | +same Pod, or against a malicious DaemonSet sharing the same node? Is |
| 342 | +that out of scope (delegated to Pod isolation / Pod Security |
| 343 | +Standards)? Proposed: out of scope. *(maps to §7, §9)* |
| 344 | + |
| 345 | +**Q9.** Meta — should this delta live at `docs/threat-model.md` in |
| 346 | +`apache/cloudstack-kubernetes-provider`, or in the website tree? |
| 347 | + |
| 348 | +**Q10.** When the main model's signing algorithm changes, what is the |
| 349 | +release-cadence commitment for an updated CCM image? |
| 350 | + |
| 351 | +**Q11.** Confirm the unsupported-component list (tests, dev scripts, |
| 352 | +old in-tree references in the README). |
| 353 | + |
| 354 | +**Q12.** TCP-Proxy LoadBalancer support pulls in HAProxy PROXY |
| 355 | +protocol *(documented: `README.md`)*. Is the CCM responsible for any |
| 356 | +authentication of the PROXY-protocol header, or is that downstream of |
| 357 | +the actual workload Pod? Proposed: downstream of the workload Pod. |
0 commit comments