worked demo with multiple phala machines

Author: amiller

tutorial/05-onchain-oracle/NOTES.md

@@ -0,0 +1,63 @@
+# Implementation Notes: Self-Join Oracle
+
+Notes from implementing multi-node oracle deployment with shared signing keys.
+
+## Challenges Encountered
+
+### 1. CLI `--custom-app-id` Flag is Disabled
+
+The `--custom-app-id` flag exists in the CLI's help text but **the implementation is commented out** in the phala-cloud-cli source (`src/commands/deploy/index.ts` lines 126-162):
+
+```typescript
+// TODO: remove customAppId for now
+// if (customAppId) { ... }  // ALL COMMENTED OUT
+```
+
+Every CLI deployment creates a new AppAuth contract, making self-join impossible via CLI alone. This is why we use direct API calls in `deploy_with_contract.py` and `deploy_replica.py`.
+
+### 2. API Key Decryption
+
+The CLI encrypts stored API keys (`~/.phala-cloud/api-key`) with AES-256-CBC using a machine-specific key:
+
+```python
+parts = f"{hostname}|{platform}|{arch}|{cpu_model}|{username}"
+key = hashlib.sha256(parts.encode()).digest()
+```
+
+**Gotcha**: Python `platform.machine()` returns `x86_64`, but Node.js `os.arch()` returns `x64`. The scripts include this conversion.
+
+### 3. allowAnyDevice Defaults to False
+
+First replica failed at "requesting app keys" because the AppAuth contract only allowed the original device (prod5). Required redeploying with `allowAnyDevice=true` via direct contract call.
+
+### 4. compose_hash Includes CVM Name
+
+Each CVM name produces a different compose_hash. Replicas fail with "Compose hash not allowed" until you call `addComposeHash()` on the AppAuth contract for each replica's hash.
+
+**Workaround**: After deploying a replica, get its compose_hash and register it:
+
+```python
+from web3 import Web3
+w3 = Web3(Web3.HTTPProvider("https://mainnet.base.org"))
+app_auth = w3.eth.contract(address=APP_AUTH_ADDRESS, abi=APP_AUTH_ABI)
+tx = app_auth.functions.addComposeHash(compose_hash_bytes).build_transaction(...)
+```
+
+### 5. Image Version Matters
+
+Base KMS clusters showed "No available resources" with `v0.5.4-dev`. Using `dstack-0.5.4` (non-dev image) worked.
+
+## Future CLI Improvements Needed
+
+1. Re-enable `--custom-app-id` flag
+2. Add `--allow-any-device` flag for AppAuth deployment
+3. Add command to register additional compose hashes (`phala app add-compose-hash`)
+
+## Successful Deployment
+
+After resolving these issues, both oracles (prod5 and prod9) share identical:
+- appId: `5a367973f645a11328d5b80fc226e3cb7436f78e`
+- signerAddress: `0x7B83657880051cD6782E1D7fFFf3e6bd54f06853`
+- derivedPubkey: `0x0323c9da9e831c9a677a92597a95c40bd7e7fe723d215763ef70874ae5dd660404`
+
+Both produce identical signatures for the same input, enabling oracle redundancy.

tutorial/05-onchain-oracle/README.md

@@ -0,0 +1,213 @@
+# Tutorial 05: On-Chain Verifiable Oracle
+
+Build an oracle that signs price data with TEE-derived keys, verifiable on-chain. This tutorial demonstrates the **AppAuth contract** architecture that controls which TEEs can get keys from KMS.
+
+## Understanding AppAuth
+
+Every dstack app has an **AppAuth contract** on Base. When a TEE requests keys, KMS calls your AppAuth's `isAppAllowed()` to decide.
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    DstackKms Contract (Base)                 │
+│                                                             │
+│  registerApp(address) ← registers your AppAuth              │
+│  isAppAllowed(bootInfo) → delegates to your contract        │
+└─────────────────────────────────────────────────────────────┘
+                              │
+                              │ calls IAppAuth(appId).isAppAllowed(bootInfo)
+                              ▼
+┌─────────────────────────────────────────────────────────────┐
+│              Your AppAuth Contract (DstackApp)               │
+│                                                             │
+│  owner: 0x...  ← can add devices/hashes                     │
+│  allowedDeviceIds[...]                                      │
+│  allowedComposeHashes[...]                                  │
+│  allowAnyDevice: true/false                                 │
+│                                                             │
+│  isAppAllowed(bootInfo) → checks whitelist                  │
+└─────────────────────────────────────────────────────────────┘
+```
+
+**Key insight:** The private key you deploy with becomes the **owner** of the AppAuth contract. The owner can add devices and compose hashes to the whitelist.
+
+## Four Deployment Options
+
+### Option 1: Single Device (CLI Default)
+
+```bash
+phala deploy -n my-oracle -c docker-compose.yaml \
+  --kms-id kms-base-prod5 \
+  --private-key "$PRIVATE_KEY"
+```
+
+Creates AppAuth with:
+- `owner` = address derived from PRIVATE_KEY
+- `allowAnyDevice = false`
+- `allowedDeviceIds[thisDevice] = true`
+- `allowedComposeHashes[thisHash] = true`
+
+**Result:** Only this device can get keys. Simple single-node deployment.
+
+### Option 2: Owner Adds Devices (Recommended for Multi-Node)
+
+Deploy first node, then the **owner** explicitly whitelists additional devices:
+
+```bash
+# Step 1: Deploy first node (owner = your address)
+phala deploy -n my-oracle -c docker-compose.yaml \
+  --kms-id kms-base-prod5 \
+  --private-key "$PRIVATE_KEY"
+
+# Step 2: Get device ID of second node (from Phala Cloud API)
+# Step 3: Owner calls addDevice() on the AppAuth contract
+cast send $APP_AUTH_ADDRESS "addDevice(bytes32)" $DEVICE_ID_2 \
+  --private-key "$PRIVATE_KEY" --rpc-url https://mainnet.base.org
+
+# Step 4: Deploy second node with same appId
+python3 deploy_replica.py
+```
+
+**Result:** Controlled multi-node. Owner explicitly approves each device.
+
+### Option 3: allowAnyDevice (Permissive)
+
+Deploy with `allowAnyDevice=true` to let any TEE device get keys:
+
+```bash
+python3 deploy_with_contract.py  # Sets allowAnyDevice=true
+```
+
+The contract checks:
+```solidity
+function isAppAllowed(AppBootInfo calldata bootInfo) {
+    if (!allowedComposeHashes[bootInfo.composeHash])
+        return (false, "Compose hash not allowed");
+    if (!allowAnyDevice && !allowedDeviceIds[bootInfo.deviceId])
+        return (false, "Device not allowed");
+    return (true, "");
+}
+```
+
+**Result:** Any device with a whitelisted composeHash can join. Less secure but simpler.
+
+### Option 4: Custom AppAuth Contract
+
+Deploy your own contract implementing `IAppAuth`:
+
+```solidity
+interface IAppAuth {
+    function isAppAllowed(AppBootInfo calldata bootInfo)
+        external view returns (bool isAllowed, string memory reason);
+}
+```
+
+Then register it:
+```solidity
+DstackKms(KMS_ADDRESS).registerApp(yourContract);  // Public function!
+```
+
+**Examples:**
+- NFT-gated: Only NFT holders can run nodes
+- DAO-controlled: Compose hashes approved by vote
+- Time-locked: Deployments only during certain periods
+- Multi-sig: Multiple approvals required
+
+## DstackApp Owner Functions
+
+The standard AppAuth (`DstackApp`) gives the owner these functions:
+
+```solidity
+// Whitelist management
+function addDevice(bytes32 deviceId) external onlyOwner;
+function removeDevice(bytes32 deviceId) external onlyOwner;
+function addComposeHash(bytes32 composeHash) external onlyOwner;
+function removeComposeHash(bytes32 composeHash) external onlyOwner;
+
+// Mode toggle
+function setAllowAnyDevice(bool allow) external onlyOwner;
+
+// Upgrades
+function disableUpgrades() external onlyOwner;  // Permanent!
+```
+
+## Signature Chain
+
+Regardless of which option you use, the oracle proves its keys came from KMS:
+
+```
+KMS Root (known on-chain)
+    │
+    │ signs: "dstack-kms-issued:" + appId + appPubkey
+    ▼
+App Key (recovered from kmsSignature)
+    │
+    │ signs: "ethereum:" + derivedPubkeyHex
+    ▼
+Derived Key → signs oracle messages
+```
+
+## Quick Start (Local Simulator)
+
+```bash
+pip install -r requirements.txt
+phala simulator start
+
+docker compose build
+docker compose run --rm -p 8080:8080 \
+  -v ~/.phala-cloud/simulator/0.5.3/dstack.sock:/var/run/dstack.sock \
+  app
+
+# In another terminal
+python3 test_local.py
+```
+
+## Files
+
+```
+05-onchain-oracle/
+├── docker-compose.yaml        # Oracle app
+├── TeeOracle.sol              # On-chain signature verification
+├── test_local.py              # Test with simulator
+├── test_phalacloud.py         # Test on Phala Cloud
+├── deploy_with_contract.py    # Deploy with allowAnyDevice=true (Option 3)
+├── deploy_replica.py          # Deploy replica using existing appId
+├── add_device.py              # Add device to whitelist (Option 2)
+├── add_compose_hash.py        # Add compose hash to whitelist
+├── requirements.txt
+├── NOTES.md                   # Troubleshooting & CLI limitations
+└── README.md
+```
+
+## Contract Addresses
+
+| Contract | Address |
+|----------|---------|
+| DstackKms (Base) | `0x2f83172A49584C017F2B256F0FB2Dca14126Ba9C` |
+| KMS Root (Simulator) | `0x8f2cF602C9695b23130367ed78d8F557554de7C5` |
+
+## IAppAuth Interface
+
+From [dstack/kms/auth-eth/contracts/IAppAuth.sol](https://github.com/dstack-tee/dstack/blob/main/kms/auth-eth/contracts/IAppAuth.sol):
+
+```solidity
+struct AppBootInfo {
+    address appId;
+    bytes32 composeHash;
+    address instanceId;
+    bytes32 deviceId;
+    bytes32 mrAggregated;
+    bytes32 mrSystem;
+    bytes32 osImageHash;
+    string tcbStatus;
+    string[] advisoryIds;
+}
+
+function isAppAllowed(AppBootInfo calldata bootInfo)
+    external view returns (bool isAllowed, string memory reason);
+```
+
+## References
+
+- [DstackKms.sol](https://github.com/dstack-tee/dstack/blob/main/kms/auth-eth/contracts/DstackKms.sol)
+- [DstackApp.sol](https://github.com/dstack-tee/dstack/blob/main/kms/auth-eth/contracts/DstackApp.sol)
+- [NOTES.md](NOTES.md)