Skip to content

Commit 9035db0

Browse files
committed
docs: document the GPU attestation collateral proxy
Covers the motivation (issue #778: OCSP as a boot DoS), the nonce relaxation trade-off, setup via vmm.toml, and operational notes.
1 parent 401c849 commit 9035db0

1 file changed

Lines changed: 90 additions & 0 deletions

File tree

docs/gpu-attestation-proxy.md

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# GPU Attestation Collateral Proxy
2+
3+
Local GPU attestation (`nvattest --verifier local`) makes live calls to two
4+
NVIDIA services at every CVM boot: the OCSP responder
5+
(`ocsp.ndis.nvidia.com`, one request per certificate in three chains) and the
6+
RIM service (`rim.attestation.nvidia.com`, driver + VBIOS RIM documents).
7+
Because the GPU attestation gate fails closed, anything that breaks those
8+
calls — an egress-restricted or air-gapped network, an upstream outage, or a
9+
middlebox — keeps every GPU CVM from booting.
10+
11+
`gpu-attest-proxy` is a small host-side (or site-level) caching proxy that
12+
removes the guest's direct dependency on those services, in the same spirit
13+
as Intel's PCCS for TDX collateral:
14+
15+
- **OCSP** requests are relayed byte-for-byte; successful responses are
16+
cached keyed by CertID until their `nextUpdate`, so repeat boots and
17+
fleets of CVMs sharing GPUs stop hitting the responder.
18+
- **RIM** documents are cached by RIM id and re-fetched on a TTL; when the
19+
upstream is unreachable, a stale document keeps being served within a
20+
bounded window.
21+
- A background refresher renews entries before they expire, so a warm cache
22+
rides through upstream outages with close to a full validity window.
23+
24+
Everything the proxy serves is signed collateral that the guest verifies
25+
itself (OCSP response signatures and validity windows via
26+
`OCSP_basic_verify`, RIM signatures and certificate chains by the attestation
27+
SDK). The proxy needs no secrets and cannot forge responses; its worst
28+
misbehavior is withholding service, which fails closed — the same outcome as
29+
blocking NVIDIA's endpoints directly.
30+
31+
## Why the OCSP nonce check is relaxed
32+
33+
The attestation SDK adds a random nonce to every OCSP request and its
34+
built-in appraisal policy requires the response to echo it
35+
(`x-nvidia-cert-ocsp-nonce-matches`). A response cached for another request
36+
can never echo this request's nonce, so caching is only possible if the
37+
guest relaxes that single check.
38+
39+
When `gpu_attest_proxy_url` is set, `dstack-util` runs nvattest with
40+
`--relying-party-policy` pointing at NVIDIA's
41+
`allow_trust_outpost_ocsp.rego` (packaged in the image at
42+
`/usr/share/nvattest/policies/`), which keeps every built-in check except
43+
the nonce match. Freshness is then bounded by the response validity window
44+
instead of the nonce: a revoked certificate may keep passing until the
45+
cached response's `nextUpdate`. This is the same trade-off the SDK's own
46+
in-process OCSP cache makes. Guests without the proxy configured keep the
47+
default policy and the nonce check.
48+
49+
## Setup
50+
51+
Run the proxy somewhere every CVM can reach (typically on each host, next to
52+
dstack-vmm):
53+
54+
```console
55+
$ gpu-attest-proxy -c /etc/gpu-attest-proxy/gpu-attest-proxy.toml
56+
```
57+
58+
See `dstack/gpu-attest-proxy/gpu-attest-proxy.toml` for the default
59+
configuration (upstreams, cache directory, TTLs). The cache persists as one
60+
JSON file per entry under `cache_dir`, so restarts keep it warm.
61+
62+
Point guests at it through `vmm.toml`:
63+
64+
```toml
65+
[cvm]
66+
gpu_attest_proxy_url = "http://<host-bridge-ip>:8090"
67+
```
68+
69+
The VMM passes the URL to the guest via sys-config. At boot, `dstack-util`
70+
derives the SDK endpoints from it:
71+
72+
- `--ocsp-url {base}/ocsp`
73+
- `--rim-url {base}` (the SDK appends `/v1/rim/{rim_id}` itself)
74+
- `--relying-party-policy /usr/share/nvattest/policies/allow_trust_outpost_ocsp.rego`
75+
76+
When the value is empty (default), guests talk to NVIDIA directly and the
77+
built-in appraisal policy applies unchanged.
78+
79+
## Operational notes
80+
81+
- `GET /health` and `GET /info` expose liveness and cache statistics.
82+
- A cold cache cannot help: the first boot after an outage begins still
83+
needs a reachable upstream. Keep the proxy running so entries stay warm;
84+
`refresh_margin` controls how far ahead of expiry entries are renewed.
85+
- The guest-side clock check is unchanged: OCSP validity windows are
86+
verified by the guest, so an expired cached response never passes.
87+
- Revocation visibility degrades from "immediate" (nonce) to the OCSP
88+
response validity window (`nextUpdate`). Measure the actual window for
89+
your certificate chains — it determines both how long an outage the cache
90+
can absorb and how long a revocation can go unnoticed.

0 commit comments

Comments
 (0)