Merge branch 'dev' of github.com:MorpheusAIs/Morpheus-Lumerin-Node into absgrafx/main

Author: alex-sandrk

.ai-docs/TEE_Attestation_Architecture.md

@@ -1,10 +1,10 @@
 # CI/CD Supply-Chain Hardening for Morpheus Docker Images
 
-**Last updated:** 2026-04-22
+**Last updated:** 2026-04-29
 **First successful run (Phase 1a — signing):** [#22920492249](https://github.com/MorpheusAIs/Morpheus-Lumerin-Node/actions/runs/22920492249)
 **First end-to-end run (Phase 1b — deploy + verify):** [#22969993910](https://github.com/MorpheusAIs/Morpheus-Lumerin-Node/actions/runs/22969993910)
 
-> **v7.0.0 release status.** The CI/CD hardening described here is the foundation that every downstream trust check depends on. Both **Phase 1c** (consumer-side proxy-router verification of the P-Node) and **Phase 2** (P-Node verifies its own backend LLM) have shipped on top of it — see [`TEE_Attestation_Architecture.md`](TEE_Attestation_Architecture.md) §7.4 and §7.7 for the code-level flow. The CI/CD pipeline itself remains unchanged from Phase 1b in this release; v7 is the *consumer* of the artifacts this pipeline produces.
+> **v7.x release status (with AMD SEV-SNP).** The CI/CD hardening described here is the foundation that every downstream trust check depends on. Both **Phase 1c** (consumer-side proxy-router verification of the P-Node) and **Phase 2** (P-Node verifies its own backend LLM) have shipped on top of it — see [`TEE_Attestation_Architecture.md`](TEE_Attestation_Architecture.md) §7.4 and §7.7. **As of this release the pipeline now also publishes AMD SEV-SNP launch-digest goldens (one per portal template), pins SecretVM `v0.0.27-alpha.1` for both TDX and SEV rootfs, and bakes the proxy-router's privacy-sensitive `LOG_LEVEL_*` fields into the cosign-signed manifest's `baked_env` block.** The new SEV path mirrors the TDX one — Python compute script in CI (`compute-sev-measurement.py`) backed by the runtime Go source-of-truth (`sev_gctx.go`) with a parity test gating drift between the two.
 
 ---
 
@@ -90,24 +90,25 @@ The SBOM is attached to the image in GHCR as an OCI artifact and travels with th
 
 ### 4. TEE Attestation Manifest
 
-This is the most important new artifact. It's a signed JSON document that records everything needed to verify a TEE deployment:
+This is the most important new artifact. It's a signed JSON document that records everything needed to verify a TEE deployment, on **both** Intel TDX and AMD SEV-SNP platforms:
 
 ```json
 {
-  "tee_image": "ghcr.io/morpheusais/morpheus-lumerin-node-tee:v5.14.7-tee-supply-chain",
+  "tee_image": "ghcr.io/morpheusais/morpheus-lumerin-node-tee@sha256:DIGEST",
+  "tee_image_tag": "ghcr.io/morpheusais/morpheus-lumerin-node-tee:vX.Y.Z-branch",
   "tee_image_digest": "sha256:3bc2f2f9...",
-  "base_image": "ghcr.io/morpheusais/morpheus-lumerin-node:v5.14.7-tee-supply-chain",
+  "base_image": "ghcr.io/morpheusais/morpheus-lumerin-node:vX.Y.Z-branch",
   "base_image_digest": "sha256:67dbc859...",
-  "compose_file": "proxy-router/docker-compose.tee.yml",
   "compose_sha256": "sha256:9b4b4fce...",
+  "compose_image_reference": "ghcr.io/.../morpheus-lumerin-node-tee@sha256:DIGEST",
   "dockerfile_tee_sha256": "sha256:30094e96...",
   "build": {
     "commit": "369e9027dc048b52003ca8abd4fbeb278196cba4",
-    "ref": "refs/heads/cicd/tee-supply-chain",
+    "ref": "refs/heads/cicd/tee-sev-and-secretvm-v0.0.27",
     "workflow": "build.yml",
     "run_id": "22920492249",
     "run_url": "https://github.com/MorpheusAIs/Morpheus-Lumerin-Node/actions/runs/22920492249",
-    "timestamp": "2026-03-10T19:46:38Z"
+    "timestamp": "2026-04-29T15:00:00Z"
   },
   "tee_platforms": ["intel-tdx", "amd-sev-snp"],
   "runtime_secrets_only": [
@@ -127,14 +128,40 @@ This is the most important new artifact. It's a signed JSON document that record
     "LOG_COLOR": "false",
     "LOG_JSON": "true",
     "LOG_IS_PROD": "true",
+    "LOG_LEVEL_APP": "info",
+    "LOG_LEVEL_TCP": "warn",
+    "LOG_LEVEL_ETH_RPC": "warn",
+    "LOG_LEVEL_STORAGE": "warn",
     "ENVIRONMENT": "production"
   },
   "measurements": {
     "intel_tdx": {
       "rtmr3": "<96-char-hex — computed from sha256(compose) + sha256(rootfs)>",
-      "secretvm_release": "v0.0.25",
-      "rootfs_variant": "rootfs-prod-tdx",
-      "rootfs_sha256": "<sha256 of rootfs-prod-v0.0.25-tdx.iso>"
+      "secretvm_release": "v0.0.27-alpha.1",
+      "rootfs_variant": "rootfs-prod",
+      "rootfs_sha256": "<sha256 of rootfs-prod-v0.0.27-alpha.1-tdx.iso>",
+      "note": "RTMR3 measures (compose + rootfs); template-independent. RTMR0-2 verified at runtime via SecretVM TDX artifact registry lookup."
+    },
+    "amd_sev_snp": {
+      "vcpu_type": "EPYC",
+      "vm_type": "prod",
+      "secretvm_release": "v0.0.27-alpha.1",
+      "rootfs_sha256": "<sha256 of rootfs-prod-v0.0.27-alpha.1-sev.iso>",
+      "kernel_hash":   "<sha256 of bzImage-v0.0.27-alpha.1-sev>",
+      "initrd_hash":   "<sha256 of initramfs-v0.0.27-alpha.1-sev.cpio.gz>",
+      "ovmf_hash":     "<sha384 of ovmf-v0.0.27-alpha.1-sev.fd, registry value>",
+      "kernel_cmdline": "console=ttyS0 loglevel=7 docker_compose_hash=<...> rootfs_hash=<...>",
+      "artifact_registry": {
+        "url": "https://raw.githubusercontent.com/scrtlabs/secretvm-verify/main/artifacts_registry/sev.json",
+        "sha256": "<sha256 of sev.json at build time>"
+      },
+      "per_template": {
+        "small":   "<96-char-hex SEV launch digest, vCPU=1>",
+        "medium":  "<96-char-hex SEV launch digest, vCPU=2>",
+        "large":   "<96-char-hex SEV launch digest, vCPU=4>",
+        "2xlarge": "<96-char-hex SEV launch digest, vCPU=8>",
+        "4xlarge": "<96-char-hex SEV launch digest, vCPU=16>"
+      }
     }
   }
 }
@@ -147,9 +174,14 @@ This manifest is signed with cosign and attached to the image using `cosign atte
 - **Image provenance**: Which commit, branch, workflow run, and timestamp produced this image. You can trace back to the exact source code.
 - **Image chain**: The TEE image's digest AND the base image's digest. You can verify both independently.
 - **Config integrity**: SHA256 hashes of `docker-compose.tee.yml` and `Dockerfile.tee`. If either file was modified between the build and a deployment, the hashes won't match.
-- **Baked configuration**: The exact environment variables frozen into the image. A verifier can confirm that `PROXY_STORE_CHAT_CONTEXT=false` (no chat logging) and `ENVIRONMENT=production` are immutable — not overridable at runtime. The `network` field ("mainnet" or "testnet") and corresponding blockchain values (contract addresses, chain ID, blockscout URL) are set at build time based on the branch: `main` → mainnet (Base Mainnet 8453), `test` → testnet (Base Sepolia 84532). All other hardened settings are identical across networks.
+- **Baked configuration**: The exact environment variables frozen into the image. A verifier can confirm that:
+  - `PROXY_STORE_CHAT_CONTEXT=false` (no chat logging) and `ENVIRONMENT=production` are immutable — not overridable at runtime.
+  - The four `LOG_LEVEL_*` fields (`APP=info`, `TCP=warn`, `ETH_RPC=warn`, `STORAGE=warn`) are explicit, not assumed defaults — this guarantees the running TEE image is not silently elevated to `debug`-level app logging (which could expose request/prompt payloads). The pipeline hard-fails the build if `LOG_LEVEL_APP=debug` is detected in `Dockerfile.tee`.
+  - The `network` field ("mainnet" or "testnet") and corresponding blockchain values (contract addresses, chain ID, blockscout URL) are set at build time based on the branch: `main` → mainnet (Base Mainnet 8453), `test` → testnet (Base Sepolia 84532). All other hardened settings are identical across networks.
 - **Runtime secret boundary**: Explicitly lists the 5 variables that ARE injectable at runtime. Everything else is sealed.
 - **Platform targets**: Confirms the image is built for both Intel TDX and AMD SEV-SNP TEE platforms.
+- **TDX golden**: One `rtmr3` value (template-independent). Consumers compare it byte-for-byte to the live TDX quote.
+- **SEV-SNP golden**: A `per_template` map with one launch digest per portal-selectable VM size (small/medium/large/2xlarge/4xlarge). Consumers parse the live quote's `family_id` (`<vmType>-<template>-sev`) to pick the correct entry — `attestation.GoldenValues.MatchSEVMeasurement` does the lookup. The asymmetry vs. TDX exists because the SEV launch digest folds in one VMSA page per vCPU; TDX RTMR3 does not.
 
 ---
 
@@ -267,12 +299,15 @@ This CI/CD hardening is the **foundation layer** for the full TEE attestation lo
 
 | File | Change |
 |---|---|
-| `.github/workflows/build.yml` | Cosign signing, digest capture, SBOM, attestation manifest, RTMR3 computation, auto-deploy, and post-deploy verification. Also: GitHub Actions upgraded to Node 24-compatible versions, Go version updated to 1.23.x. |
-| `.github/tee/secretvm.env` | Pins SecretVM release version, rootfs variant, URL, and SHA-256. All pipeline rootfs references derive from this file. |
+| `.github/workflows/build.yml` | Cosign signing, digest capture, SBOM, attestation manifest, RTMR3 computation, **SEV-SNP measurement compute (per template) + SEV registry fetch + baked-loglevel extraction with privacy hard-fail**, auto-deploy, and post-deploy verification. Also: GitHub Actions upgraded to Node 24-compatible versions, Go version updated to 1.25.x. |
+| `.github/tee/secretvm.env` | Pins SecretVM release version (currently `v0.0.27-alpha.1`), rootfs variant (`rootfs-prod`), **TDX rootfs URL/SHA-256 AND SEV rootfs URL/SHA-256, plus the SEV artifact registry URL and registry vm_type**. All pipeline rootfs references derive from this file. |
 | `proxy-router/scripts/compute-rtmr3.py` | Standalone RTMR3 computation script matching the `reproduce-mr` algorithm. Can be run locally for independent verification. |
-| `proxy-router/Dockerfile.tee` | Bakes immutable ENV config into the TEE image. Blockchain values (diamond, token, blockscout, chain ID) are parameterized via `ARG` with mainnet defaults; overridden via `--build-arg` for testnet builds. |
-| `proxy-router/docker-compose.tee.yml` | Canonical compose template for TEE deployment with 5 runtime secrets |
-| `docs/02.3-proxy-router-tee.md` | Provider setup and consumer verification guide |
+| `proxy-router/scripts/compute-sev-measurement.py` | **NEW** — Standalone SEV-SNP launch-digest computation script. Byte-for-byte port of `attestation/sev_gctx.go::CalcSevMeasurement`. Computes one measurement per vCPU template (small/medium/large/2xlarge/4xlarge). Parity-tested against the runtime Go via `internal/attestation/sev_python_parity_test.go`. |
+| `proxy-router/internal/attestation/sev_python_parity_test.go` | **NEW** — Hermetic regression guard: runs `compute-sev-measurement.py` as a subprocess against the v0.0.27-alpha.1 prod registry fixture and asserts byte-for-byte match against `CalcSevMeasurement` for all 5 templates. Skips gracefully when `python3` is unavailable. |
+| `proxy-router/internal/attestation/golden.go` | Renamed JSON tag `amd_sev` → `amd_sev_snp` to align with the manifest schema; added `SEVMeasurements.PerTemplate map[string]string`, `GoldenValues.SEVPerTemplate`, and the `MatchSEVMeasurement` helper that looks up the golden by template-keyed live measurement. |
+| `proxy-router/Dockerfile.tee` | Bakes immutable ENV config into the TEE image. Blockchain values are parameterized via `ARG` with mainnet defaults; overridden via `--build-arg` for testnet builds. **Logging values (`LOG_LEVEL_APP=info`, `LOG_LEVEL_TCP=warn`, `LOG_LEVEL_ETH_RPC=warn`, `LOG_LEVEL_STORAGE=warn`, plus `LOG_COLOR=false`, `LOG_JSON=true`, `LOG_IS_PROD=true`) are baked here and surfaced into the cosign-signed manifest's `baked_env` block — the privacy gate ensures the live TEE image cannot silently revert to `debug`-level app logging.** |
+| `proxy-router/docker-compose.tee.yml` | Canonical compose template for TEE deployment with 5 runtime secrets. |
+| `docs/02.3-proxy-router-tee.md` | Provider setup and consumer verification guide. |
 
 ---
 
@@ -310,14 +345,28 @@ This CI/CD hardening is the **foundation layer** for the full TEE attestation lo
 | **`GET /v1/models/attestation`** | Per-model attestation state endpoint for monitoring and forensics | **Done** — PR #699 |
 | **New env vars** | `TEE_PORTAL_URL`, `TEE_IMAGE_REPO`, `ARTIFACT_REGISTRY_URL`, `ARTIFACT_REGISTRY_REFRESH_INTERVAL` | **Done** — PR #699 |
 
+### Completed (this PR — SEV-SNP and v0.0.27-alpha.1 cutover)
+
+| Step | Description | Status |
+|---|---|---|
+| **SecretVM `v0.0.27-alpha.1` pin** | TDX + SEV rootfs URLs and SHA-256s pinned in `.github/tee/secretvm.env` (the portal currently only allows new VM provisioning on this release) | **Done** |
+| **SEV rootfs download + SHA-256 verify** | `Download SecretVM rootfs (TDX + SEV)` step pulls both ISOs; both fail-closed if the pinned SHA doesn't match | **Done** |
+| **SEV artifact registry fetch** | `Fetch SecretVM SEV artifact registry` step downloads `sev.json` from `scrtlabs/secretvm-verify` and records its SHA-256 in the manifest | **Done** |
+| **SEV measurement compute (5 templates)** | `Compute SEV-SNP measurement (per template)` step runs `compute-sev-measurement.py --all-templates` and emits per-template digests as job outputs | **Done** |
+| **Per-template SEV measurements in cosign-signed manifest** | `measurements.amd_sev_snp.per_template` block under the existing `cosign attest` flow | **Done** |
+| **Baked log-level fields in `baked_env`** | `Extract baked log levels from Dockerfile.tee` step reads the four `LOG_LEVEL_*` ENV lines plus `LOG_COLOR`/`LOG_JSON`/`LOG_IS_PROD` and writes them to the manifest. Hard-fails if any are missing or if `LOG_LEVEL_APP=debug` (privacy gate) | **Done** |
+| **Consumer parser update** | `internal/attestation/golden.go` renamed `amd_sev` → `amd_sev_snp` JSON tag, added `PerTemplate` map and `MatchSEVMeasurement` lookup helper | **Done** |
+| **Python ↔ Go parity regression test** | `internal/attestation/sev_python_parity_test.go` runs `compute-sev-measurement.py` as a subprocess and asserts identical hex output to `CalcSevMeasurement` for all 5 templates | **Done** |
+
 ### Remaining (Lower Priority / Future)
 
 | Area | Step | Status |
 |---|---|---|
 | CI/CD | Full RTMR0-2 *recomputation* in CI (today we verify RTMR0-2 by artifact-registry lookup, which is sufficient) | TODO — blocked on ACPI templates |
-| CI/CD | AMD SEV-SNP measurement via `sev-snp-measure` | TODO — TDX-only today |
+| CI/CD | Auto-deploy + post-deploy verification for an SEV test VM (TDX-only today; needs SEV-flavoured offsets, separate `SECRETVM_TEST_SEV_VM_UUID`, parallel `Deploy-SecretVM-Test-SEV` job) | TODO |
 | CI/CD | CVE scanning (Trivy/Grype) — advisory then gating | TODO |
+| CI/CD | Scheduled monitor that polls `secret-vm-build` releases AND `secretvm-verify/artifacts_registry/sev.json` for new versions, opens an issue/PR | TODO |
 | Proxy-router | Verifiable per-message signing using SecretVM TEE-bound key | Deferred to Phase 2b |
-| Proxy-router | Local in-process quote verification (remove `quote-parse` dependency on SCRT Labs) | Deferred to Phase 2b |
+| Proxy-router | Local in-process quote verification (remove `quote-parse` / `quote-parse-sev` dependency on SCRT Labs) | Deferred to Phase 2b |
 | Proxy-router | Co-located proxy-router + LLM in a single TDX VM (collapses both hops into one RTMR3) | Deferred to Phase 2b |
 | Proxy-router | NRAS alternatives for non-NVIDIA GPU vendors | Deferred to Phase 2b |

.ai-docs/TEE_CICD_Supply_Chain_Hardening.md

@@ -1,25 +1,37 @@
-# SecretVM artifact configuration for RTMR3 computation
+# SecretVM artifact configuration for RTMR3 (Intel TDX) + SEV measurement (AMD SEV-SNP)
 # Update these when SCRT Labs publishes a new stable VM release.
 #
-# Source: https://github.com/scrtlabs/secret-vm-build/releases
-# Algorithm: scrtlabs/reproduce-mr (RTMR3 = replayRTMR of sha256(compose) + sha256(rootfs))
+# Source:    https://github.com/scrtlabs/secret-vm-build/releases
+# Algorithm: scrtlabs/reproduce-mr (Intel TDX RTMR3 = replayRTMR of sha256(compose) + sha256(rootfs))
+#            scrtlabs/secretvm-verify (AMD SEV-SNP launch digest via GCTX page-update chain)
 #
 # IMPORTANT: Always use the "prod" rootfs variant — SecretVM runs "environment prod"
-# even for developer portal deployments. Using "dev" rootfs produces wrong RTMR3.
+# even for developer portal deployments. Using "dev" rootfs produces wrong measurements.
+#
+# Compatibility note (2026-04-29):
+#   The SecretVM portal currently only allows starting *new* VMs on
+#   v0.0.27-alpha.1. Older releases (e.g. v0.0.25) can still run already-provisioned
+#   VMs but cannot be selected for new deployments.
 
-SECRETVM_RELEASE=v0.0.25
-SECRETVM_ROOTFS_VARIANT=rootfs-prod-tdx
+SECRETVM_RELEASE=v0.0.27-alpha.1
+SECRETVM_ROOTFS_VARIANT=rootfs-prod
 
 # Intel TDX rootfs (non-GPU, prod — used for proxy-router P-Node deployments)
-# URL is derived: .../download/${RELEASE}/${VARIANT_BASE}-${RELEASE}-tdx.iso
-SECRETVM_ROOTFS_TDX_URL=https://github.com/scrtlabs/secret-vm-build/releases/download/v0.0.25/rootfs-prod-v0.0.25-tdx.iso
-# SHA256 will be captured on first CI/CD run and pinned here
-SECRETVM_ROOTFS_TDX_SHA256=2a6d16965f5ce5143bcba614bb7896cdfc598a71c6e1ecc55efd3d8f74ba7c8a
+# URL pattern: .../download/${RELEASE}/${VARIANT}-${RELEASE}-tdx.iso
+SECRETVM_ROOTFS_TDX_URL=https://github.com/scrtlabs/secret-vm-build/releases/download/v0.0.27-alpha.1/rootfs-prod-v0.0.27-alpha.1-tdx.iso
+SECRETVM_ROOTFS_TDX_SHA256=1d638b068bc6f9254e74ea917818d4e09a7ffce8e486d54c5b5dbe9e6c44b261
+
+# AMD SEV-SNP rootfs (prod) — same compose, different platform binary
+# URL pattern: .../download/${RELEASE}/${VARIANT}-${RELEASE}-sev.iso
+SECRETVM_ROOTFS_SEV_URL=https://github.com/scrtlabs/secret-vm-build/releases/download/v0.0.27-alpha.1/rootfs-prod-v0.0.27-alpha.1-sev.iso
+SECRETVM_ROOTFS_SEV_SHA256=f44141c9a0cbed19ddf30b16929de33633e5631cfd68731e9ff9e4321d5775fd
 
-# AMD SEV rootfs (prod)
-# SECRETVM_ROOTFS_SEV_URL=https://github.com/scrtlabs/secret-vm-build/releases/download/v0.0.25/rootfs-prod-v0.0.25-sev.iso
-# SECRETVM_ROOTFS_SEV_SHA256=
+# SEV artifact registry — kernel/initrd/ovmf hashes + ovmf_sections for measurement compute.
+# Pulled from scrtlabs/secretvm-verify; pipeline picks the entry whose `vm_type` and
+# `artifacts_ver` match SECRETVM_ROOTFS_VARIANT (prod) and SECRETVM_RELEASE.
+SECRETVM_SEV_REGISTRY_URL=https://raw.githubusercontent.com/scrtlabs/secretvm-verify/main/artifacts_registry/sev.json
+SECRETVM_SEV_REGISTRY_VM_TYPE=prod
 
 # GPU variant (for co-located proxy+LLM deployments)
-# SECRETVM_ROOTFS_GPU_TDX_URL=https://github.com/scrtlabs/secret-vm-build/releases/download/v0.0.25/rootfs-gpu-prod-v0.0.25-tdx.iso
-# SECRETVM_ROOTFS_GPU_TDX_SHA256=
+# SECRETVM_ROOTFS_GPU_TDX_URL=https://github.com/scrtlabs/secret-vm-build/releases/download/v0.0.27-alpha.1/rootfs-gpu-prod-v0.0.27-alpha.1-tdx.iso
+# SECRETVM_ROOTFS_GPU_TDX_SHA256=ddf4b55edd296b13f046e4d515f5547c7e39da5afdea1bc1b71e23e645fe7a8a