Commit 1eec08c
authored
fix: plaintext password issue via sealed secrets (#495)
* feat: migrate secrets to SOPS encryption
- Add SOPS + Age encryption for Helm chart secrets (datamate, label-studio, milvus)
- Remove plaintext passwords from values.yaml files
- Replace hardcoded secrets in docker-compose.yml with env vars
- Add .env.example template for Docker deployment path
- Add scripts/secrets.sh helper for encrypt/decrypt/helm-install
- Add docs/SECRETS_SETUP.md setup guide
- Update .gitignore: exclude .sops-keys/, allow .env.example and .env.enc
Deployment:
- K8s/Helm: helm secrets install -f secrets.yaml
- Docker: cp .env.example .env && edit && docker compose up
* feat: update Makefile for SOPS secret management
- K8s install: use helm secrets upgrade with -f secrets.yaml for datamate, label-studio, milvus
- Docker install: add pre-check for .env file, exit with helpful message if missing
- Set SOPS_AGE_KEY_FILE env var for helm secrets decryption
* fix: remove hardcoded JWT secret and unify property naming
- JwtUtils.java: remove hardcoded default "datamate-secret-key-for-jwt-token-generation"
Change property from jwt.secret to datamate.jwt.secret (aligned with JwtConfig)
- application.yml: add datamate.jwt.secret mapping from JWT_SECRET env var
- application.yml: remove hardcoded defaults for DB_PASSWORD and REDIS_PASSWORD
- docker-compose.yml: add JWT_SECRET env var to datamate-backend service
- Helm values.yaml: add JWT_SECRET secretKeyRef to backend env
- Helm secrets.yaml: add JWT_SECRET to public.secrets.data
* fix: enforce encrypted private key and restrict permissions
- Reject plaintext private keys on startup (exit with error)
- Require CERT_PASS when key is encrypted
- Set chmod 600 on decrypted key for restrictive access
* fix: remove hardcoded passwords in Python config and exclude dev .env from Docker builds
- Create .dockerignore to prevent runtime/datamate-python/.env from being copied
into Docker images (it contained localhost:15432 telepresence debug settings)
- config.py: remove hardcoded defaults for pgsql_password, mysql_password,
label_studio_password, label_studio_user_token
- docker-compose.yml: add explicit PGSQL_HOST/PGSQL_PORT for backend-python
- Helm values.yaml: add explicit PGSQL_HOST/PGSQL_PORT for backend-python
* refactor: encapsulate SOPS in scripts/secrets.sh, remove helm-secrets dependency
- scripts/secrets.sh: add check_tools(), ensure_key() with auto-generation
New helm-upgrade command: decrypts secrets.yaml and runs helm upgrade --install
- Makefile: K8s install targets now call "bash scripts/secrets.sh helm-upgrade"
instead of direct "helm secrets upgrade". No helm-secrets plugin needed.
- Users only need sops + age (brew install), key auto-generated on first run.
Docker users unaffected - still use .env file.
* fix: pass Label Studio credentials to backend-python container
- docker-compose.yml: add LABEL_STUDIO_USER_TOKEN, LABEL_STUDIO_PASSWORD env vars
- Helm values.yaml: add secretKeyRef for both to backend-python env
- secrets.yaml: add encrypted LABEL_STUDIO_USER_TOKEN, LABEL_STUDIO_PASSWORD
Fixes 500 error "Label Studio API token is required" when creating annotation
tasks after config.py defaults were removed.
* fix: add LABEL_STUDIO_USERNAME env var for auto-login to Label Studio
The loginAnnotationUsingGet flow requires label_studio_username to auto-login
to Label Studio. Without it, clicking "edit" on an annotation task redirects
to Label Studio login page instead of auto-authenticating.
* docs: add LABEL_STUDIO_USERNAME to .env.example
* fix: remove duplicate case blocks in scripts/secrets.sh causing syntax error
* fix: add LABEL_STUDIO_USERNAME to Helm backend-python env
The loginAnnotationUsingGet API requires label_studio_username to auto-login
to Label Studio. Docker got this in 638931b, but the Helm values.yaml was
missing it, causing K8s deployments to show Label Studio login prompt.
* feat: migrate from SOPS to Sealed Secrets for K8s secret management
Replace SOPS + Age encryption with Bitnami Sealed Secrets.
No key distribution needed - secrets are encrypted with cluster public key,
decrypted automatically by the in-cluster controller.
Changes:
- Add 3 SealedSecret YAMLs (datamate, label-studio, milvus) under
deployment/kubernetes/sealed-secrets/ (safe to commit to Git)
- Update Makefile: apply SealedSecrets before helm install
- Helm charts: support existingSecret for label-studio,
secrets.create flag to skip Helm-managed Secret creation
- Remove SOPS artifacts: .sops.yaml, scripts/secrets.sh, secrets.yaml files
User workflow: make install INSTALLER=k8s
No tools needed on user machine. Controller decrypts automatically.
* fix: pgbouncer also reads POSTGRE_PASSWORD from existingSecret
When existingSecret is configured, the pgbouncer sidecar was still
reading POSTGRE_PASSWORD from .Values.env (empty after SOPS migration),
causing "password authentication failed" when connecting to PostgreSQL.
* fix: set secrets.create: false to prevent Helm-SealedSecret conflict
Helm must not create the datamate-conf Secret since it is managed
by the SealedSecret controller. Without this, fresh installs fail with
"conflict with controller" error.
* fix: gateway reads JWT_SECRET from datamate-conf Secret
Gateway was still reading JWT_SECRET from values.yaml (empty string),
causing "JWT secret is required" startup error. Changed to secretKeyRef
to match the SealedSecret-managed datamate-conf Secret.
* fix: set created_by='system' for seed data to work with JWT data scope filter
When DATAMATE_JWT_ENABLE=true, the Python backend's _apply_data_scope
filter adds WHERE created_by IN ('user', 'system') to all BaseEntity
queries. Seed data had empty created_by, causing templates, annotation
templates, and operator categories to return 0 results.
- t_clean_template: add created_by/updated_by='system' to INSERT
- t_dm_annotation_templates: add COALESCE fallback in ON CONFLICT
- t_operator_category: add UPDATE to fix empty created_by
All three tables also get a safety-net UPDATE at the end.
* fix: remove hardcoded DB_PASSWORD default fallback in gateway config
The gateway's application.yml had password: ${DB_PASSWORD:password} which
falls back to 'password' when DB_PASSWORD env var is not set. Removed the
default so it fails fast if the variable is missing, consistent with the
main-application config.
* docs: add Sealed Secrets setup guide for online and offline environments
- Add Secret Management section to README-zh.md and README.md
- Document Sealed Secrets Controller installation (Helm)
- Add air-gapped/offline environment instructions (image download)
- Add kubeseal CLI usage for updating secrets
- Add make download-sealed-secrets target for offline image download1 parent 6673572 commit 1eec08c
26 files changed
Lines changed: 530 additions & 44 deletions
File tree
- backend
- api-gateway/src/main/resources
- services/main-application/src/main/resources
- deployment
- docker/datamate
- helm
- datamate
- charts/public
- templates
- label-studio
- templates
- milvus
- kubernetes/sealed-secrets
- docs
- runtime/datamate-python/app/core
- scripts
- db
- images/frontend
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
191 | 191 | | |
192 | 192 | | |
193 | 193 | | |
194 | | - | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| 49 | + | |
49 | 50 | | |
50 | 51 | | |
51 | 52 | | |
| |||
69 | 70 | | |
70 | 71 | | |
71 | 72 | | |
| 73 | + | |
72 | 74 | | |
73 | 75 | | |
74 | 76 | | |
| |||
254 | 256 | | |
255 | 257 | | |
256 | 258 | | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
257 | 266 | | |
258 | 267 | | |
259 | 268 | | |
| |||
326 | 335 | | |
327 | 336 | | |
328 | 337 | | |
| 338 | + | |
329 | 339 | | |
330 | 340 | | |
331 | 341 | | |
332 | 342 | | |
333 | 343 | | |
334 | 344 | | |
335 | | - | |
| 345 | + | |
| 346 | + | |
336 | 347 | | |
337 | 348 | | |
338 | 349 | | |
339 | 350 | | |
340 | 351 | | |
| 352 | + | |
341 | 353 | | |
342 | 354 | | |
343 | 355 | | |
| |||
471 | 483 | | |
472 | 484 | | |
473 | 485 | | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
474 | 497 | | |
475 | 498 | | |
476 | 499 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
35 | | - | |
| 35 | + | |
36 | 36 | | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
37 | 79 | | |
38 | 80 | | |
39 | 81 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
40 | 82 | | |
41 | 83 | | |
42 | 84 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
25 | | - | |
| 25 | + | |
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| |||
Lines changed: 6 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
| 17 | + | |
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
62 | | - | |
| 62 | + | |
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
| |||
131 | 131 | | |
132 | 132 | | |
133 | 133 | | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
134 | 138 | | |
135 | 139 | | |
136 | 140 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
| 19 | + | |
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
| 12 | + | |
| 13 | + | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
| |||
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
33 | | - | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
34 | 37 | | |
35 | 38 | | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
36 | 42 | | |
37 | 43 | | |
38 | 44 | | |
| |||
52 | 58 | | |
53 | 59 | | |
54 | 60 | | |
55 | | - | |
| 61 | + | |
56 | 62 | | |
57 | 63 | | |
58 | 64 | | |
| |||
75 | 81 | | |
76 | 82 | | |
77 | 83 | | |
78 | | - | |
| 84 | + | |
79 | 85 | | |
80 | 86 | | |
81 | 87 | | |
| |||
93 | 99 | | |
94 | 100 | | |
95 | 101 | | |
96 | | - | |
| 102 | + | |
97 | 103 | | |
98 | 104 | | |
99 | 105 | | |
| |||
213 | 219 | | |
214 | 220 | | |
215 | 221 | | |
216 | | - | |
| 222 | + | |
217 | 223 | | |
218 | 224 | | |
219 | 225 | | |
| |||
241 | 247 | | |
242 | 248 | | |
243 | 249 | | |
244 | | - | |
| 250 | + | |
245 | 251 | | |
246 | 252 | | |
247 | 253 | | |
248 | 254 | | |
249 | 255 | | |
250 | 256 | | |
251 | 257 | | |
252 | | - | |
| 258 | + | |
253 | 259 | | |
254 | | - | |
| 260 | + | |
255 | 261 | | |
256 | 262 | | |
257 | 263 | | |
| |||
290 | 296 | | |
291 | 297 | | |
292 | 298 | | |
293 | | - | |
294 | | - | |
| 299 | + | |
| 300 | + | |
295 | 301 | | |
296 | 302 | | |
297 | 303 | | |
| |||
0 commit comments