|
| 1 | +# Bug: v2 actor dispatch requires ~5s delay after metadata refresh |
| 2 | + |
| 3 | +## Reproduce |
| 4 | + |
| 5 | +```bash |
| 6 | +# 1. Start engine with the force-v2 hack (see below) |
| 7 | +rm -rf ~/.local/share/rivet-engine/db |
| 8 | +cargo run --bin rivet-engine -- start |
| 9 | + |
| 10 | +# 2. Start test-envoy |
| 11 | +RIVET_ENDPOINT=http://127.0.0.1:6420 RIVET_TOKEN=dev RIVET_NAMESPACE=default \ |
| 12 | + RIVET_POOL_NAME=test-envoy AUTOSTART_ENVOY=0 AUTOSTART_SERVER=1 \ |
| 13 | + AUTOCONFIGURE_SERVERLESS=0 cargo run -p rivet-test-envoy |
| 14 | + |
| 15 | +# 3. In another terminal, run this: |
| 16 | +NS="repro-$(date +%s)" |
| 17 | +curl -s -X POST -H "Authorization: Bearer dev" -H "Content-Type: application/json" \ |
| 18 | + http://localhost:6420/namespaces -d "{\"name\":\"$NS\",\"display_name\":\"$NS\"}" |
| 19 | +curl -s -X PUT -H "Authorization: Bearer dev" -H "Content-Type: application/json" \ |
| 20 | + "http://localhost:6420/runner-configs/test-envoy?namespace=$NS" \ |
| 21 | + -d '{"datacenters":{"default":{"serverless":{"url":"http://localhost:5051/api/rivet","request_lifespan":300,"max_concurrent_actors":10000,"slots_per_runner":1,"min_runners":0,"max_runners":10000}}}}' |
| 22 | +curl -s -X POST -H "Authorization: Bearer dev" -H "Content-Type: application/json" \ |
| 23 | + "http://localhost:6420/runner-configs/test-envoy/refresh-metadata?namespace=$NS" -d '{}' |
| 24 | + |
| 25 | +# THIS FAILS (no delay): |
| 26 | +curl -s -X POST -H "Authorization: Bearer dev" -H "Content-Type: application/json" \ |
| 27 | + "http://localhost:6420/actors?namespace=$NS" \ |
| 28 | + -d "{\"name\":\"test\",\"key\":\"k-$(date +%s)\",\"runner_name_selector\":\"test-envoy\",\"crash_policy\":\"sleep\"}" \ |
| 29 | + | python3 -c "import json,sys; a=json.load(sys.stdin)['actor']['actor_id']; print(a)" \ |
| 30 | + | xargs -I{} curl -s --max-time 12 -H "X-Rivet-Token: dev" -H "X-Rivet-Target: actor" -H "X-Rivet-Actor: {}" http://localhost:6420/ping |
| 31 | +# Expected: actor_ready_timeout |
| 32 | + |
| 33 | +# THIS WORKS (5s delay): |
| 34 | +sleep 5 |
| 35 | +curl -s -X POST -H "Authorization: Bearer dev" -H "Content-Type: application/json" \ |
| 36 | + "http://localhost:6420/actors?namespace=$NS" \ |
| 37 | + -d "{\"name\":\"test\",\"key\":\"k2-$(date +%s)\",\"runner_name_selector\":\"test-envoy\",\"crash_policy\":\"sleep\"}" \ |
| 38 | + | python3 -c "import json,sys; a=json.load(sys.stdin)['actor']['actor_id']; print(a)" \ |
| 39 | + | xargs -I{} curl -s --max-time 12 -H "X-Rivet-Token: dev" -H "X-Rivet-Target: actor" -H "X-Rivet-Actor: {}" http://localhost:6420/ping |
| 40 | +# Expected: 200 with JSON body |
| 41 | +``` |
| 42 | + |
| 43 | +## Symptom |
| 44 | + |
| 45 | +Actor is created (200), envoy receives CommandStartActor, actor starts in ~10ms, EventActorStateUpdate{Running} is sent back via WS, but the guard returns `actor_ready_timeout` after 10 seconds. The actor never becomes connectable. |
| 46 | + |
| 47 | +## Root cause |
| 48 | + |
| 49 | +After `refresh-metadata` stores `envoyProtocolVersion` in the DB, the runner pool workflow (`pegboard_runner_pool`) needs to restart its serverless connection cycle to use v2 POST instead of v1 GET. This takes ~2-5 seconds because: |
| 50 | + |
| 51 | +1. The `pegboard_runner_pool_metadata_poller` workflow runs on a polling interval |
| 52 | +2. The `pegboard_serverless_conn` workflow needs to cycle its existing connections |
| 53 | +3. The `pegboard_runner_pool` workflow reads the updated config and spawns new v2 connections |
| 54 | + |
| 55 | +Until this happens, the engine dispatches via v1 GET SSE which doesn't deliver the start payload to the envoy. |
| 56 | + |
| 57 | +## Code locations |
| 58 | + |
| 59 | +### Force-v2 hack (temporary) |
| 60 | +`engine/packages/pegboard/src/workflows/actor/runtime.rs` line ~268: |
| 61 | +```rust |
| 62 | +// Changed from: if pool.and_then(|p| p.protocol_version).is_some() |
| 63 | +// To force v2 for all serverless pools: |
| 64 | +if pool.as_ref().and_then(|p| p.protocol_version).is_some() || for_serverless { |
| 65 | +``` |
| 66 | + |
| 67 | +### Where protocol_version is stored |
| 68 | +`engine/packages/pegboard/src/workflows/runner_pool_metadata_poller.rs` line ~214: |
| 69 | +```rust |
| 70 | +if let Some(protocol_version) = metadata.envoy_protocol_version { |
| 71 | + tx.write(&protocol_version_key, protocol_version)?; |
| 72 | +} |
| 73 | +``` |
| 74 | + |
| 75 | +### Where protocol_version is read for v1→v2 migration decision |
| 76 | +`engine/packages/pegboard/src/workflows/actor/runtime.rs` in `allocate_actor_v2`: |
| 77 | +```rust |
| 78 | +let pool_res = ctx.op(crate::ops::runner_config::get::Input { ... }).await?; |
| 79 | +// ... |
| 80 | +if pool.and_then(|p| p.protocol_version).is_some() { |
| 81 | + return Ok(AllocateActorOutputV2 { status: AllocateActorStatus::MigrateToV2, ... }); |
| 82 | +} |
| 83 | +``` |
| 84 | + |
| 85 | +### Where runner config is cached (may need invalidation) |
| 86 | +`engine/packages/pegboard/src/ops/runner_config/get.rs` - reads ProtocolVersionKey from DB |
| 87 | + |
| 88 | +### Where v1 (GET) vs v2 (POST) connection is made |
| 89 | +- v1: `engine/packages/pegboard/src/workflows/serverless/conn.rs` line ~301: `client.get(endpoint_url)` |
| 90 | +- v2: `engine/packages/pegboard-outbound/src/lib.rs` line ~316: `client.post(endpoint_url).body(payload)` |
| 91 | + |
| 92 | +## Fix needed |
| 93 | + |
| 94 | +After `refresh-metadata` stores `envoyProtocolVersion`, the runner pool should immediately use v2 POST without waiting for the metadata poller cycle. Either: |
| 95 | +1. Signal the runner pool workflow to restart connections when metadata changes |
| 96 | +2. Make the `refresh-metadata` API synchronously update the runner pool state |
| 97 | +3. Have the serverless conn workflow check protocol_version before each connection attempt instead of relying on the metadata poller cycle |
0 commit comments