vishnu2kmohan · vishnu2kmohan · Dec 8, 2025 · Dec 8, 2025
@@ -5,7 +5,7 @@
 [![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
 [![Production Ready](https://img.shields.io/badge/production-ready-brightgreen.svg)](docs/deployment/production-checklist.mdx)
 [![Documentation](https://img.shields.io/badge/docs-mintlify-green.svg)](https://vishnu2kmohan.github.io/mcp-server-langgraph/)
-[![ADRs](https://img.shields.io/badge/ADRs-63-informational.svg)](adr/README.md)
+[![ADRs](https://img.shields.io/badge/ADRs-64-informational.svg)](adr/README.md)
 [![Use This Template](https://img.shields.io/badge/use-this%20template-blue.svg?logo=cookiecutter)](https://github.com/vishnu2kmohan/mcp-server-langgraph#-use-this-template)
 [![Docker](https://img.shields.io/badge/docker-%230db7ed.svg?logo=docker&logoColor=white)](Dockerfile)
 [![Kubernetes](https://img.shields.io/badge/kubernetes-%23326ce5.svg?logo=kubernetes&logoColor=white)](docs/deployment/kubernetes.mdx)

@@ -1,7 +1,7 @@
 # Architecture Decision Records (ADRs)
 
-**Last Updated**: 2025-11-30
-**Total ADRs**: 63
+**Last Updated**: 2025-12-07
+**Total ADRs**: 64
 
 ## Overview
 
@@ -110,6 +110,7 @@ Each ADR follows this structure:
 | [ADR-0041](adr-0041-postgresql-gdpr-storage.md) | 41. Pure PostgreSQL for GDPR/HIPAA/SOC2 Compliance Storage | Accepted | 2025-11-02 |
 | [ADR-0056](adr-0056-asyncmock-configuration-prevention.md) | 56. AsyncMock Configuration Prevention Mechanisms | Accepted | 2025-11-13 |
 | [ADR-0066](adr-0066-helm-chart-security-risk-acceptance.md) | 66. Helm Chart Security Risk Acceptance | Accepted | 2025-11-29 |
+| [ADR-0067](adr-0067-non-root-container-security.md) | Non-Root Container Security Strategy | Accepted | 2025-12-07 |
 
 ### Testing & Quality
 

@@ -0,0 +1,68 @@
+# ADR-0067: Non-Root Container Security Strategy
+
+**Status**: Accepted
+**Date**: 2025-12-07
+**Author**: Claude Code
+**Decision Makers**: Vishnu Mohan
+
+## Context
+
+During CI troubleshooting for E2E tests, we discovered that several containers in `docker-compose.test.yml` were running as root or had file permission issues that caused failures in CI but not locally. This highlighted the need for a comprehensive non-root container security strategy.
+
+## Decision
+
+### 1. Non-Root by Default
+
+All containers MUST run as non-root users unless there is a documented, justified exception.
+
+### 2. UID/GID Strategy
+
+Use the UID specified by the upstream container image maintainer:
+
+| Container | UID | Image Source |
+|-----------|-----|--------------|
+| Loki | 10001 | grafana/loki |
+| Prometheus | 65534 (nobody) | prom/prometheus |
+| Alertmanager | 65534 (nobody) | prom/alertmanager |
+| Grafana | 472 | grafana/grafana |
+| Jaeger | 10001 | jaegertracing/all-in-one |
+| Keycloak | 1000 | quay.io/keycloak/keycloak |
+| Qdrant | 1000 | qdrant/qdrant:*-unprivileged |
+| Traefik | 1000 | traefik:v3.x (with high ports) |
+| PostgreSQL | 70 | postgres:*-alpine |
+| Redis | 999 | redis:*-alpine |
+
+### 3. tmpfs Ownership
+
+When using tmpfs mounts, explicitly set UID/GID:
+
+```yaml
+tmpfs:
+  - /path:rw,noexec,nosuid,size=256m,uid=10001,gid=10001
+```
+
+### 4. Documented Exceptions
+
+**Promtail** is the only container allowed to run as root because:
+- Requires read access to Docker socket
+- Needs to read container log files owned by various UIDs
+
+## Consequences
+
+### Positive
+
+- Security parity: Local, CI, and production environments use consistent non-root policies
+- Earlier failure detection: Permission issues caught locally instead of CI
+- OpenShift ready: Containers work with OpenShift's restrictive SCC
+- Reduced attack surface: Compromised containers cannot easily escalate privileges
+
+### Negative
+
+- Minor complexity: Must track different UIDs for different images
+- Port restrictions: Services requiring port below 1024 need port mapping
+
+## References
+
+- [Docker Rootless Mode](https://docs.docker.com/engine/security/rootless/)
+- [Kubernetes Security Context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)
+- [OpenShift Container Security](https://docs.openshift.com/container-platform/latest/authentication/managing-security-context-constraints.html)
@@ -67,25 +67,33 @@ services:
   # Dashboard available at http://localhost/gateway or :8080
 
   traefik-gateway:
-    image: traefik:v3.2
+    # v3.6: Latest stable with Gateway API v1.3+ support and security fixes
+    image: traefik:v3.6
+    # Run as non-root user (requires high ports internally)
+    # Reference: https://community.traefik.io/t/running-traefik-as-non-root-in-a-container/16381
+    user: "1000:1000"
     command:
       # API and Dashboard
       - "--api.dashboard=true"
       - "--api.insecure=true"
+      - "--ping=true"
       # Docker provider
       - "--providers.docker=true"
       - "--providers.docker.exposedbydefault=false"
       - "--providers.docker.network=mcp-test-network"
-      # Entrypoints
-      - "--entrypoints.web.address=:80"
+      # Entrypoints - use high ports (>1024) for non-root
+      - "--entrypoints.web.address=:8000"
+      - "--entrypoints.dashboard.address=:8081"
       # Logging
       - "--log.level=INFO"
       - "--accesslog=true"
       - "--accesslog.format=json"
     ports:
-      - "80:80"      # Main gateway (single entry point)
-      - "8080:8080"  # Traefik dashboard
+      - "80:8000"    # Main gateway (external 80 -> internal 8000)
+      - "8080:8081"  # Traefik dashboard (external 8080 -> internal 8081)
     volumes:
+      # Docker socket requires root:docker group. Container user 1000 needs
+      # docker group membership on host, or use docker-socket-proxy for production
       - /var/run/docker.sock:/var/run/docker.sock:ro
     healthcheck:
       test: ["CMD", "traefik", "healthcheck", "--ping"]
@@ -118,8 +126,7 @@ services:
   # ==============================================================================
 
   postgres-test:
-    # PostgreSQL 16 to match CloudSQL (terraform/environments/gcp-preview/main.tf: postgres_major_version = 16)
-    image: postgres:16-alpine
+    image: postgres:16.8-alpine
     environment:
       POSTGRES_USER: postgres
       POSTGRES_PASSWORD: postgres
@@ -280,7 +287,7 @@ services:
   # ==============================================================================
 
   redis-test:
-    # Redis 7.2 to match Memorystore (terraform/environments/gcp-preview/main.tf: redis_version = "REDIS_7_2")
+    # Use 7.2 to match GCP Memorystore supported version for local/cloud parity
     image: redis:7.2-alpine
     command: redis-server --appendonly no --maxmemory 128mb --maxmemory-policy allkeys-lru --save ""
     ports:
@@ -311,13 +318,17 @@ services:
   # ==============================================================================
 
   qdrant-test:
-    image: qdrant/qdrant:v1.15.5
+    # Use unprivileged variant for non-root security (UID 1000)
+    # Reference: https://qdrant.tech/documentation/guides/security/
+    image: qdrant/qdrant:v1.15.5-unprivileged
+    user: "1000:1000"
     ports:
       - "9333:6333"  # HTTP API
       - "9334:6334"  # gRPC API
     # No volumes - ephemeral storage
     tmpfs:
-      - /qdrant/storage:rw,noexec,nosuid,size=256m
+      # UID 1000 is the Qdrant user in the unprivileged image
+      - /qdrant/storage:rw,noexec,nosuid,size=256m,uid=1000,gid=1000
     environment:
       - QDRANT_ALLOW_RECOVERY_MODE=true
       - QDRANT__LOG_LEVEL=WARN  # Reduced logging
@@ -630,14 +641,16 @@ services:
   # Logs accessible at /logs or via Grafana Explore
 
   loki-test:
-    image: grafana/loki:3.0.0
+    # v3.6.2: Latest stable with improved performance and query capabilities
+    image: grafana/loki:3.6.2
     command: -config.file=/etc/loki/local-config.yaml
     ports:
       - "13100:3100"  # Loki API (offset by 10000)
     volumes:
       - ./docker/loki/loki-config.yaml:/etc/loki/local-config.yaml:ro
     tmpfs:
-      - /loki:rw,noexec,nosuid,size=256m
+      # UID 10001 is the loki user in grafana/loki:3.0.0
+      - /loki:rw,noexec,nosuid,size=256m,uid=10001,gid=10001
     healthcheck:
       test: ["CMD", "wget", "--spider", "-q", "http://localhost:3100/ready"]
       interval: 10s
@@ -666,7 +679,8 @@ services:
       - "traefik.http.routers.loki.middlewares=loki-strip"
 
   promtail-test:
-    image: grafana/promtail:3.0.0
+    # v3.6.2: Match Loki version (Promtail deprecated in favor of Grafana Alloy)
+    image: grafana/promtail:3.6.2
     command: -config.file=/etc/promtail/config.yaml
     volumes:
       - ./docker/promtail/promtail-config.yaml:/etc/promtail/config.yaml:ro

@@ -74,18 +74,17 @@ ENV UV_COMPILE_BYTECODE=1 \
     UV_PYTHON_DOWNLOADS=never \
     UV_PROJECT_ENVIRONMENT=/app/.venv
 
-# Copy dependency files
+# Copy dependency files and source code
+# NOTE: Source must be copied BEFORE uv sync because pyproject.toml references src/
 COPY pyproject.toml uv.lock ./
+COPY src/ ./src/
 
 # Install Python dependencies (production only, no dev extras)
 # --frozen: Use lockfile exactly, fail if out of sync
 # --no-dev: Skip development dependencies
 RUN --mount=type=cache,target=/root/.cache/uv \
     uv sync --frozen --no-dev
 
-# Copy source code
-COPY src/ ./src/
-
 # Copy built frontend from stage 1
 # Place in builder/frontend/dist to match SPAStaticFiles path expectations
 COPY --from=frontend-builder /frontend/dist ./src/mcp_server_langgraph/builder/frontend/dist/

@@ -63,8 +63,10 @@ ENV UV_COMPILE_BYTECODE=1 \
     UV_PYTHON_DOWNLOADS=never \
     UV_PROJECT_ENVIRONMENT=/app/.venv
 
-# Copy dependency files
+# Copy dependency files and source code
+# NOTE: Source must be copied BEFORE uv sync because pyproject.toml references src/
 COPY pyproject.toml uv.lock ./
+COPY src/ ./src/
 
 # Install Python dependencies with playground extras
 # --frozen: Use lockfile exactly, fail if out of sync
@@ -73,9 +75,6 @@ COPY pyproject.toml uv.lock ./
 RUN --mount=type=cache,target=/root/.cache/uv \
     uv sync --frozen --no-dev --extra playground
 
-# Copy source code
-COPY src/ ./src/
-
 # Create non-root user for security (DS002 compliance)
 RUN useradd -m -u 1000 appuser && \
     chown -R appuser:appuser /app

@@ -0,0 +1,148 @@
+---
+title: "ADR-0067: Non-Root Container Security Strategy"
+description: "Architecture Decision Record: ADR-0067: Non-Root Container Security Strategy"
+icon: 'shield-halved'
+---
+
+# ADR-0067: Non-Root Container Security Strategy
+
+**Status**: Accepted
+**Date**: 2025-12-07
+**Author**: Claude Code
+**Decision Makers**: Vishnu Mohan
+
+## Context
+
+During CI troubleshooting for E2E tests, we discovered that several containers in `docker-compose.test.yml` were running as root or had file permission issues that caused failures in CI but not locally. This highlighted the need for a comprehensive non-root container security strategy that:
+
+1. Aligns local development with CI and production Kubernetes deployments
+2. Supports OpenShift's Security Context Constraints (SCC)
+3. Follows container security best practices
+4. Documents exceptions where root is required
+
+### Problem Statement
+
+- **Loki container** crashed with "permission denied" on config file and tmpfs storage
+- **Qdrant** and **Traefik** were running as root, inconsistent with Kubernetes policies
+- Local development didn't catch these issues due to different file ownership contexts
+- No documented strategy for container UID/GID selection
+
+## Decision
+
+### 1. Non-Root by Default
+
+All containers MUST run as non-root users unless there is a documented, justified exception.
+
+### 2. UID/GID Strategy
+
+Use the UID specified by the upstream container image maintainer rather than standardizing on a single UID:
+
+| Container | UID | Image Source |
+|-----------|-----|--------------|
+| Loki | 10001 | grafana/loki |
+| Prometheus | 65534 (nobody) | prom/prometheus |
+| Alertmanager | 65534 (nobody) | prom/alertmanager |
+| Grafana | 472 | grafana/grafana |
+| Jaeger | 10001 | jaegertracing/all-in-one |
+| Keycloak | 1000 | quay.io/keycloak/keycloak |
+| Qdrant | 1000 | qdrant/qdrant:*-unprivileged |
+| Traefik | 1000 | traefik:v3.x (with high ports) |
+| PostgreSQL | 70 | postgres:*-alpine |
+| Redis | 999 | redis:*-alpine |
+
+### 3. tmpfs Ownership
+
+When using tmpfs mounts for ephemeral storage, explicitly set UID/GID ownership:
+
+```yaml
+tmpfs:
+  - /path:rw,noexec,nosuid,size=256m,uid=10001,gid=10001
+```
+
+### 4. Config File Permissions
+
+All mounted config files MUST have mode 644 (world-readable) to support containers running as non-root:
+
+```bash
+# Correct: World-readable
+-rw-r--r-- loki-config.yaml
+
+# Incorrect: Owner-only (breaks non-root containers)
+-rw------- loki-config.yaml
+```
+
+### 5. Privileged Port Handling
+
+For services requiring ports < 1024, use internal high ports with port mapping:
+
+```yaml
+traefik-gateway:
+  user: "1000:1000"
+  command:
+    - "--entrypoints.web.address=:8000"  # High port internally
+  ports:
+    - "80:8000"  # External 80 -> internal 8000
+```
+
+### 6. Documented Exceptions
+
+**Promtail** is the only container allowed to run as root because:
+- Requires read access to Docker socket (`/var/run/docker.sock`)
+- Needs to read container log files owned by various UIDs
+- No practical non-root alternative without losing functionality
+
+### 7. OpenShift Compatibility
+
+For OpenShift deployments, containers must support arbitrary UIDs with GID 0:
+
+```dockerfile
+# Set ownership to root group (GID 0) for OpenShift compatibility
+RUN chown -R 1000:0 /app && chmod -R g=u /app
+```
+
+OpenShift's `restricted` SCC assigns random UIDs at runtime but preserves GID 0 membership.
+
+## Consequences
+
+### Positive
+
+- **Security parity**: Local, CI, and production environments use consistent non-root policies
+- **Earlier failure detection**: Permission issues caught locally instead of CI
+- **OpenShift ready**: Containers work with OpenShift's restrictive SCC
+- **Reduced attack surface**: Compromised containers cannot easily escalate privileges
+- **Compliance**: Meets SOC 2, HIPAA, and PCI-DSS container security requirements
+
+### Negative
+
+- **Minor complexity**: Must track different UIDs for different images
+- **Port restrictions**: Services requiring port below 1024 need port mapping configuration
+- **Image selection**: Must use `-unprivileged` variants when available (e.g., Qdrant)
+- **File permissions**: Config files must be world-readable (acceptable for non-secret configs)
+
+### Neutral
+
+- **No performance impact**: Non-root execution has negligible overhead
+- **Promtail exception**: Documented and accepted for log collection functionality
+
+## Implementation
+
+### Completed (PR #144)
+
+1. ✅ Loki: tmpfs with `uid=10001,gid=10001`
+2. ✅ Qdrant: Changed to `qdrant/qdrant:v1.15.5-unprivileged` with `user: "1000:1000"`
+3. ✅ Traefik: High ports (8000/8081) with `user: "1000:1000"`
+4. ✅ Config file permissions documented
+
+### Future Work
+
+1. Update Dockerfiles for OpenShift GID 0 compatibility
+2. Add pre-commit hook to validate container UIDs in docker-compose files
+3. Document in `deployments/README.md`
+
+## References
+
+- [Docker Rootless Mode](https://docs.docker.com/engine/security/rootless/)
+- [Kubernetes Security Context](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)
+- [OpenShift Container Security](https://docs.openshift.com/container-platform/latest/authentication/managing-security-context-constraints.html)
+- [Qdrant Unprivileged Image](https://qdrant.tech/documentation/guides/security/)
+- [Traefik Non-Root Guide](https://community.traefik.io/t/running-traefik-as-non-root-in-a-container/16381)