Skip to content

Commit 12ee26f

Browse files
h4x3rotabclaude
h4x3rotab
and
claude
committed
docs: add GPU TEE deployment guide for agents
Add .agent/GPU_TEE_DEPLOYMENT.md with learnings from deploying GPU workloads to Phala Cloud: - Instance types (tdx.* for CPU, h200.* for GPU) - Docker Compose GPU configuration - vLLM and vllm-proxy setup - Debugging commands and common issues 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 666720f commit 12ee26f

4 files changed

Lines changed: 244 additions & 0 deletions

File tree

‎.agent/GPU_TEE_DEPLOYMENT.md‎

Lines changed: 235 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,235 @@
1+
# GPU TEE Deployment Guide
2+
3+
Learnings from deploying GPU workloads to Phala Cloud TEE infrastructure.
4+
5+
## Instance Types
6+
7+
Query available instance types:
8+
```bash
9+
curl -s "https://cloud-api.phala.network/api/v1/instance-types" | jq
10+
```
11+
12+
### CPU-only (Intel TDX)
13+
- `tdx.small` through `tdx.8xlarge`
14+
15+
### GPU (H200 + TDX)
16+
- `h200.small` — Single H200 GPU, suitable for inference
17+
- `h200.16xlarge` — Multi-GPU for larger workloads
18+
- `h200.8x.large` — High-memory configuration
19+
20+
## Deployment Commands
21+
22+
### GPU Deployment
23+
```bash
24+
phala deploy -n my-app -c docker-compose.yaml \
25+
--instance-type h200.small \
26+
--region US-EAST-1 \
27+
--image dstack-nvidia-dev-0.5.4.1
28+
```
29+
30+
Key flags:
31+
- `--instance-type h200.small` — Required for GPU access
32+
- `--image dstack-nvidia-dev-0.5.4.1` — NVIDIA development image with GPU drivers
33+
- `--region US-EAST-1` — Region with GPU nodes (gpu-use2)
34+
35+
### Debugging
36+
```bash
37+
# Check CVM status
38+
phala cvms list
39+
40+
# View serial logs (boot + container output)
41+
phala cvms serial-logs <app_id> --tail 100
42+
43+
# Delete CVM
44+
phala cvms delete <name-or-id> --force
45+
```
46+
47+
## Docker Compose GPU Configuration
48+
49+
GPU devices must be explicitly reserved in docker-compose.yaml:
50+
51+
```yaml
52+
services:
53+
my-gpu-app:
54+
image: my-image
55+
deploy:
56+
resources:
57+
reservations:
58+
devices:
59+
- driver: nvidia
60+
count: all
61+
capabilities: [gpu]
62+
```
63+
64+
Without the `deploy.resources.reservations.devices` section, the container will fail with:
65+
```
66+
libcuda.so.1: cannot open shared object file: No such file or directory
67+
```
68+
69+
## vLLM Example
70+
71+
Working docker-compose.yaml for vLLM inference:
72+
73+
```yaml
74+
services:
75+
vllm:
76+
image: vllm/vllm-openai:latest
77+
volumes:
78+
- /var/run/dstack.sock:/var/run/dstack.sock
79+
environment:
80+
- NVIDIA_VISIBLE_DEVICES=all
81+
- HF_TOKEN=${HF_TOKEN:-}
82+
ports:
83+
- "8000:8000"
84+
command: >
85+
--model Qwen/Qwen2.5-1.5B-Instruct
86+
--host 0.0.0.0
87+
--port 8000
88+
--max-model-len 4096
89+
--gpu-memory-utilization 0.8
90+
deploy:
91+
resources:
92+
reservations:
93+
devices:
94+
- driver: nvidia
95+
count: all
96+
capabilities: [gpu]
97+
```
98+
99+
## Endpoint URLs
100+
101+
After deployment, the app is accessible at:
102+
```
103+
https://<app_id>-<port>.dstack-pha-<region>.phala.network
104+
```
105+
106+
Example for vLLM on port 8000:
107+
```bash
108+
# List models
109+
curl https://<app_id>-8000.dstack-pha-use2.phala.network/v1/models
110+
111+
# Chat completion
112+
curl -X POST https://<app_id>-8000.dstack-pha-use2.phala.network/v1/chat/completions \
113+
-H "Content-Type: application/json" \
114+
-d '{"model": "Qwen/Qwen2.5-1.5B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
115+
```
116+
117+
## vllm-proxy (Response Signing)
118+
119+
vllm-proxy provides response signing and attestation for vLLM inference. It sits between clients and vLLM, signing responses with TEE-derived keys.
120+
121+
### Configuration
122+
123+
**IMPORTANT**: The authentication environment variable is `TOKEN`, not `AUTH_TOKEN`.
124+
125+
```yaml
126+
services:
127+
vllm:
128+
image: vllm/vllm-openai:latest
129+
environment:
130+
- NVIDIA_VISIBLE_DEVICES=all
131+
command: >
132+
--model Qwen/Qwen2.5-1.5B-Instruct
133+
--host 0.0.0.0
134+
--port 8000
135+
--max-model-len 4096
136+
--gpu-memory-utilization 0.8
137+
deploy:
138+
resources:
139+
reservations:
140+
devices:
141+
- driver: nvidia
142+
count: all
143+
capabilities: [gpu]
144+
145+
proxy:
146+
image: phalanetwork/vllm-proxy:v0.2.18
147+
volumes:
148+
- /var/run/dstack.sock:/var/run/dstack.sock # Required for TEE key derivation
149+
environment:
150+
- VLLM_BASE_URL=http://vllm:8000
151+
- MODEL_NAME=Qwen/Qwen2.5-1.5B-Instruct
152+
- TOKEN=your-secret-token # NOT AUTH_TOKEN
153+
ports:
154+
- "8000:8000"
155+
depends_on:
156+
- vllm
157+
```
158+
159+
### API Endpoints
160+
161+
```bash
162+
# List models (no auth required)
163+
curl https://<endpoint>/v1/models
164+
165+
# Chat completion (requires auth)
166+
curl -X POST https://<endpoint>/v1/chat/completions \
167+
-H "Content-Type: application/json" \
168+
-H "Authorization: Bearer your-secret-token" \
169+
-d '{"model": "Qwen/Qwen2.5-1.5B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
170+
171+
# Get response signature
172+
curl https://<endpoint>/v1/signature/<chat_id> \
173+
-H "Authorization: Bearer your-secret-token"
174+
175+
# Attestation report
176+
curl https://<endpoint>/v1/attestation/report \
177+
-H "Authorization: Bearer your-secret-token"
178+
```
179+
180+
### Tested Configuration
181+
182+
- Image: `phalanetwork/vllm-proxy:v0.2.18`
183+
- Instance: `h200.small`
184+
- Region: `US-EAST-1`
185+
- Model: `Qwen/Qwen2.5-1.5B-Instruct`
186+
187+
### vllm-proxy Issues
188+
189+
**"Invalid token" error**:
190+
- Check that you're using `TOKEN` environment variable, not `AUTH_TOKEN`
191+
- Verify the token value matches your request header
192+
193+
**"All connection attempts failed" from proxy**:
194+
- vLLM is still loading the model (takes 1-2 minutes after container starts)
195+
- Wait for vLLM to show "Uvicorn running on" in serial logs
196+
197+
**NVML error on attestation**:
198+
- GPU confidential computing attestation may not be fully available
199+
- This doesn't affect inference or response signing
200+
201+
## Common Issues
202+
203+
### "No available resources match your requirements"
204+
- GPU nodes are limited. Wait for other CVMs to finish or try a different region.
205+
- Ensure you're using the correct instance type (`h200.small`).
206+
207+
### Container crashes with GPU errors
208+
- Add `deploy.resources.reservations.devices` section to docker-compose.yaml.
209+
- Verify using NVIDIA development image (`dstack-nvidia-dev-*`).
210+
211+
### Image pull takes too long
212+
- Large images (5GB+ for vLLM) take 3-5 minutes to download and extract.
213+
- Check serial logs for progress.
214+
215+
## Testing Workflow
216+
217+
1. Deploy: `phala deploy -n test -c docker-compose.yaml --instance-type h200.small --region US-EAST-1 --image dstack-nvidia-dev-0.5.4.1`
218+
2. Wait for status: `phala cvms list` (wait for "running")
219+
3. Check logs: `phala cvms serial-logs <app_id> --tail 100`
220+
4. Test API: `curl https://<app_id>-<port>.dstack-pha-use2.phala.network/...`
221+
5. Cleanup: `phala cvms delete <name> --force`
222+
223+
## GPU Wrapper Script
224+
225+
For repeated GPU deployments, use a wrapper script:
226+
227+
```bash
228+
#!/bin/bash
229+
# phala-gpu.sh
230+
source "$(dirname "$0")/.env"
231+
export PHALA_CLOUD_API_KEY=$PHALA_CLOUD_API_GPU
232+
phala "$@"
233+
```
234+
235+
This allows maintaining separate API keys for CPU and GPU workspaces.

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,4 @@ node_modules/
99
/tmp
1010
.claude/settings.local.json
1111
__pycache__
12+
.planning/

‎CLAUDE.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -229,3 +229,4 @@ When need more detailed info, try to use deepwiki mcp.
229229

230230
The `.agent/` directory contains AI assistant resources:
231231
- `WRITING_GUIDE.md` — Documentation and README writing guidelines (messaging, style, audiences)
232+
- `GPU_TEE_DEPLOYMENT.md` — GPU deployment to Phala Cloud (instance types, docker-compose config, debugging)

‎README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,13 @@ Apps communicate with the guest agent via HTTP over `/var/run/dstack.sock`. Use
108108
- [Design Decisions](./docs/design-and-hardening-decisions.md) - Architecture rationale
109109
- [FAQ](./docs/faq.md) - Frequently asked questions
110110

111+
## Trusted by
112+
113+
- [OpenRouter](https://openrouter.ai/provider/phala) - Confidential AI inference
114+
- [NEAR AI](https://x.com/ilblackdragon/status/1962920246148268235) - Private AI agents
115+
116+
dstack is a [Confidential Computing Consortium](https://confidentialcomputing.io/2025/10/02/welcoming-phala-to-the-confidential-computing-consortium/) open source project.
117+
111118
## Community
112119

113120
[Telegram](https://t.me/+UO4bS4jflr45YmUx) · [GitHub Discussions](https://github.com/Dstack-TEE/dstack/discussions) · [Examples](https://github.com/Dstack-TEE/dstack-examples)

0 commit comments

Comments
 (0)