test(provisioner): real-backend gRPC round-trips for Mongo/Queue/Storage (Wave 4) (#47)

mastermanas805 · claude · web-flow · commit 69ad7f27cb4c · 2026-06-04T17:20:33.000Z
* test(provisioner): real-backend gRPC round-trips for Mongo/Queue/Storage (Wave 4) Closes the remaining cells of the provisioner gRPC × backend matrix (INTEGRATION-COVERAGE-PLAN §2.3 / Wave 4). Postgres + Redis server-layer round-trips already shipped (#45, #46); this adds the real-backend Provision → assert artifact → Deprovision → assert-gone lifecycle for the three remaining backends, all driven through the genuine gRPC handlers (breaker wrapping, tier routing, mapError, response shaping): - Mongo: ProvisionResource creates usr_/db_ on a real MongoDB, GetStorageBytes reads real dbStats (>0 after seeding), DeprovisionResource runs the real dropUser/dropDatabase (truehomie DROP-incident class), second Deprovision is a clean idempotent no-op, and Regrade(mongo) asserts the documented skip path. - Queue (NATS): ProvisionResource passes the real NATS monitor health check and returns nats:// URL + subject prefix, GetStorageBytes(queue)=0 (message-metered), Deprovision is the shared-backend no-op, idempotent. - Storage (MinIO/S3): GetStorageBytes object-walk — empty prefix=0, after a real PutObject=exact byte count, after delete=0. (Storage Provision/Deprovision are API-side; provisioner only meters.) All tests env-gated (skip cleanly under `go test -short`, the deploy gate; run for real when the backend env is present). CI: added NATS service container + MinIO docker-run step + the TEST_NATS_HOST / TEST_MINIO_* / CUSTOMER_MONGO_AUTH_URL env wiring to coverage.yml so they execute (mongo was already provided). Verified locally against real mongo/nats/minio containers: all 8 server round-trip tests PASS; integration-only coverage for internal/server = 99.2% (provisionMongo/provisionQueue/GetStorageBytes/DeprovisionResource/RegradeResource all 100%). No bug found in the destroy/regrade paths. Added INTEGRATION-COVERAGE-EXCLUSIONS.md documenting the ≥80 floor method + the only genuinely-unreachable lines (k8s dedicated-backend boot wiring, cmd/ entrypoints). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci(provisioner): start NATS via docker run, not a services container The minimal nats:2 image has no wget/curl/nc, so the service-container --health-cmd ('wget ... :8222/healthz') could never pass — GitHub Actions marked the container unhealthy and aborted the coverage job before any test ran (NATS logged 'Server is ready'). Mirror the MinIO pattern: docker run nats:2 -js -m 8222 + a runner-side curl wait on /healthz. Unblocks #47. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Manas Srivastava <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml
@@ -101,6 +101,22 @@ jobs:
           --health-timeout 5s
           --health-retries 10
 
+      # NATS (JetStream + HTTP monitor :8222) for the server-layer queue
+      # round-trip (internal/server/server_live_roundtrip_mqs_test.go::
+      # TestServer_Queue_*) is started as a `docker run` step below (see
+      # "Start NATS for queue round-trip"), NOT a `services:` container: the
+      # minimal nats:2 image has no wget/curl/nc, so a service-container
+      # `--health-cmd` can NEVER pass — GitHub Actions then marks the container
+      # unhealthy and aborts the whole job even though the server logged
+      # "Server is ready". docker-run + a runner-side curl wait avoids that.
+
+      # NOTE: MinIO is NOT a `services:` container — GitHub Actions service
+      # containers cannot supply the required `server /data` command (services
+      # accept image/env/ports/options only, not args). MinIO is started as a
+      # `docker run` step below (see "Start MinIO for storage round-trip"), which
+      # the server-layer storage test (TestServer_Storage_GetStorageBytes_*)
+      # reaches at 127.0.0.1:9100.
+
     # TEST_* env vars the pool/backend test helpers read via os.Getenv. The
     # exact names were grepped from the test files (os.Getenv("TEST_…") and the
     # liveXxx() helper fallbacks) — wrong names = silent skip = lost coverage.
@@ -120,7 +136,19 @@ jobs:
       # internal/backend/mongo/local_test.go::liveMongoURI
       CUSTOMER_MONGO_URL: mongodb://127.0.0.1:27017
       # internal/backend/mongo/coverage_extra_test.go auth-fail branches
+      # ALSO: internal/server/server_live_roundtrip_mqs_test.go::liveMongoAdminURI
+      # (the gRPC-layer mongo Provision/StorageBytes/Deprovision round-trip uses
+      # the authenticated instance — the realistic prod createUser path).
       CUSTOMER_MONGO_AUTH_URL: mongodb://root:rootpass@127.0.0.1:27018
+      # internal/server/server_live_roundtrip_mqs_test.go::liveNATSHost — the
+      # queue LocalBackend health-checks http://$NATS_HOST:8222/healthz.
+      TEST_NATS_HOST: 127.0.0.1
+      # internal/server/server_live_roundtrip_mqs_test.go::liveStorageEndpoint —
+      # the storage GetStorageBytes round-trip against the MinIO started below.
+      TEST_MINIO_ENDPOINT: 127.0.0.1:9100
+      TEST_MINIO_ROOT_USER: minioadmin
+      TEST_MINIO_ROOT_PASSWORD: minioadmin
+      TEST_MINIO_BUCKET: itest-bucket
 
     steps:
       - uses: actions/checkout@v6
@@ -146,6 +174,46 @@ jobs:
       # ≠ verified). A red test must fail the job; codecov upload below is
       # still soft-failed via fail_ci_if_error: false so a missing token
       # doesn't take CI down.
+      # MinIO can't run as a `services:` container (no way to pass `server /data`).
+      # Start it here so TestServer_Storage_GetStorageBytes_LiveRoundTrip has a
+      # real S3-compatible endpoint at 127.0.0.1:9100. Env (MINIO_ROOT_USER/
+      # PASSWORD) are static literals — no untrusted workflow input is interpolated.
+      - name: Start MinIO for storage round-trip
+        run: |
+          docker run -d --name itest-minio \
+            -p 9100:9000 \
+            -e MINIO_ROOT_USER=minioadmin \
+            -e MINIO_ROOT_PASSWORD=minioadmin \
+            minio/minio:latest server /data
+          # Wait for MinIO to report healthy before the test run.
+          for i in $(seq 1 30); do
+            if curl -fsS http://127.0.0.1:9100/minio/health/live >/dev/null 2>&1; then
+              echo "minio healthy after ${i} tries"; break
+            fi
+            sleep 1
+          done
+          curl -fsS http://127.0.0.1:9100/minio/health/live >/dev/null
+
+      # NATS can't be a `services:` container (minimal nats:2 image has no
+      # wget/curl for a passing --health-cmd). Start it here with explicit
+      # -js (JetStream) + -m 8222 (HTTP monitor) and wait on /healthz from the
+      # runner, which DOES have curl. Reached by the queue round-trip at
+      # 127.0.0.1:4222 / :8222 (TEST_NATS_HOST=127.0.0.1).
+      - name: Start NATS for queue round-trip
+        run: |
+          docker run -d --name itest-nats \
+            -p 4222:4222 \
+            -p 8222:8222 \
+            nats:2 -js -m 8222
+          # Wait for the NATS HTTP monitor to report healthy before the tests.
+          for i in $(seq 1 30); do
+            if curl -fsS http://127.0.0.1:8222/healthz >/dev/null 2>&1; then
+              echo "nats healthy after ${i} tries"; break
+            fi
+            sleep 1
+          done
+          curl -fsS http://127.0.0.1:8222/healthz >/dev/null
+
       - name: Generate coverage
         working-directory: provisioner
         # No `-short`: the integration tests are gated on TEST_* env-var
diff --git a/INTEGRATION-COVERAGE-EXCLUSIONS.md b/INTEGRATION-COVERAGE-EXCLUSIONS.md
@@ -0,0 +1,50 @@
+# Provisioner — Integration-Coverage Exclusions
+
+> Companion to the standing integration-coverage program
+> (`docs/sessions/2026-06-04/INTEGRATION-COVERAGE-PLAN-2026-06-04.md`, Wave 4).
+>
+> The provisioner's forum-defined done-bar is a **≥80% line-coverage floor measured
+> integration-only** (real backends, not mocks) AND flow-completeness (every gRPC
+> Provision / Deprovision / Status / Regrade flow has a real-backend round-trip).
+>
+> The ≥80 floor is computed **after subtracting the lines listed here** — each is
+> genuinely unreachable from an integration test (process bootstrap / k8s
+> control-plane wiring / fatal-exit) and would otherwise force a fake just to
+> tick a line. Every entry has a one-line justification. Keep this list *small* —
+> it is exclusions, not waivers; a flow that can be driven against a real backend
+> must be, not listed here.
+
+## How integration-only coverage is measured (mechanism C, per the PLAN)
+
+```bash
+# from provisioner/, with real backends reachable (CI coverage.yml supplies them):
+#   Postgres, Redis, Mongo (no-auth + auth), NATS (monitor :8222), MinIO.
+export CUSTOMER_MONGO_AUTH_URL=mongodb://root:rootpass@127.0.0.1:27018
+export TEST_NATS_HOST=127.0.0.1
+export TEST_MINIO_ENDPOINT=127.0.0.1:9100
+export TEST_MINIO_ROOT_USER=minioadmin TEST_MINIO_ROOT_PASSWORD=minioadmin TEST_MINIO_BUCKET=itest-bucket
+export TEST_POSTGRES_CUSTOMERS_URL=postgres://postgres:postgres@127.0.0.1:5432/postgres?sslmode=disable
+export CUSTOMER_REDIS_URL=redis://127.0.0.1:6379
+
+go test ./internal/server/ -coverpkg=./internal/server/... \
+  -coverprofile=/tmp/srvcov.out -count=1 -timeout 300s
+go tool cover -func=/tmp/srvcov.out | tail -1
+```
+
+Measured for `internal/server` (the gRPC handler layer — the package this Wave-4
+PR touched), all backends wired: **99.2%** of statements — well above the 80
+floor before any subtraction.
+
+## Excluded lines (genuinely unreachable from an integration test)
+
+| Location | Symbol | Why it cannot be integration-covered |
+|---|---|---|
+| `internal/server/server.go` `New()` — the `cfg.K8sEnabled` block | dedicated-backend init (`postgres/redis/mongo/queue.NewK8sDedicatedBackend`) | Requires a live kube-apiserver + kubeconfig; the dedicated (Pro/Team) k8s backends are exercised by the per-backend `k8s_live_test.go` suites against a real cluster (nightly e2e), not by the in-process gRPC server round-trip. The error-log fallback branches fire only on a malformed kubeconfig at process boot. |
+| `internal/server/server.go` `NewWithBackends()` — `poolMgr != nil` typed-nil normalization | constructor guard | The branch that converts a typed-nil `*pool.Manager` to a nil `PoolClaimer` interface is a boot-time correctness guard; it is covered by the unit constructor test, but the "real pool manager passed" arm needs a live `*pgxpool.Pool` to the provisioner DB and is exercised by `internal/pool/manager_db_test.go`, not the server round-trip. |
+| `cmd/` (all) | `main`, signal wiring, `os.Exit` | Process entrypoint / fatal-exit paths — not reachable without forking the binary. Excluded from the measured package set (PLAN §1.4). |
+
+Everything else in the gRPC handler layer — every `ProvisionResource`,
+`DeprovisionResource`, `GetStorageBytes`, `RegradeResource` arm across postgres,
+redis, mongo, queue, and storage — IS driven by a real-backend round-trip
+(`server_live_roundtrip_test.go` + `server_live_roundtrip_mqs_test.go`) and sits
+at 100% function coverage.
diff --git a/internal/server/server_live_roundtrip_mqs_test.go b/internal/server/server_live_roundtrip_mqs_test.go
