Intermittent 'unexpected message from server' with tokio-postgres query_typed on Hyperdrive

[yonatan-genai]

Description

When using tokio-postgres with Hyperdrive via workers-rs, issuing multiple query_typed calls on the same Client connection intermittently fails with "unexpected message from server". The error is non-deterministic — the same query succeeds sometimes and fails others, with roughly 30-50% failure rate on repeated requests.

Environment

• workers-rs 0.8.x on wasm32-unknown-unknown
• tokio-postgres 0.7.x with query_typed (unnamed statement / extended query protocol)
• Hyperdrive with caching enabled, transaction-mode pooling
• Supabase Postgres origin (PostgreSQL 15)

Reproduction

1. Open a single TCP socket to Hyperdrive via Socket::builder().secure_transport(SecureTransport::StartTls)
2. connect_raw with tokio_postgres::Config from hyperdrive.connection_string()
3. Execute a query_typed call — succeeds
4. Execute a second query_typed call on the same Client — intermittently fails with "unexpected message from server"

The error rate increases with query complexity (larger SQL strings fail more often) but even simple queries fail sometimes.

Key finding: protocol mixing

Using simple_query (simple query protocol) followed by query_typed (extended query protocol) on the same connection reliably produces the error. This suggests Hyperdrive's message buffering/replay logic desynchronizes when it sees different protocol message types on the same client socket.

Workaround

Opening a fresh TCP socket to Hyperdrive for each query operation eliminates the issue. This is consistent with the docs' guidance that "creating a new client on each invocation is fast and recommended" — but that guidance appears to be about JS clients where each operation naturally creates a new socket. For Rust/tokio-postgres, a Client is designed to be reused across multiple queries on the same connection, and the extended query protocol expects consistent wire state.

The workaround works but adds ~1-2ms per query (edge-local socket setup) and defeats Hyperdrive's connection multiplexing within a single request handler that needs multiple queries.

Expected behavior

Multiple query_typed calls on the same Hyperdrive connection should work reliably, as they do with a direct connection to PostgreSQL. The extended query protocol (Parse → Bind → Describe → Execute → Sync) should maintain consistent state across queries on the same socket.

Hypothesis

Hyperdrive's transaction-mode pooling routes individual queries within a single client socket to different backend PostgreSQL connections. When tokio-postgres sends the second query's Parse message, the backend connection may have been swapped, and the response doesn't match what the client's protocol state machine expects. The prepared statements blog post describes Hyperdrive buffering and replaying messages — the replay logic may not correctly handle the state transitions of the unnamed-statement extended query protocol used by query_typed.

Docs gap

The Connect to PostgreSQL docs list rust-postgres v0.19.8 as supported and say "Use the query_typed method for best performance" but don't mention:

• Whether a single Client can issue multiple queries (answer: apparently not reliably)
• That mixing simple and extended query protocols is unsupported
• That a fresh socket per query is required for reliable operation

Adding this guidance would save Rust/Workers developers significant debugging time.

[yonatan-genai]

Update: Root cause is query caching, not connection pooling.

After further testing, the issue is specifically Hyperdrive's query caching (caching.disabled: false). With caching enabled and max_age: 5, complex queries (3.4KB CTEs with window functions) intermittently fail. With caching.disabled: true, the exact same queries succeed 10/10 every time.

Simpler queries (single-table SELECT with WHERE clause) work fine even with caching enabled. The failure correlates with query complexity/size.

This means the per-query fresh connection workaround was masking a different issue — the query cache was returning corrupted results, which tokio-postgres interpreted as protocol state errors.

Reproduction updated:

1. Create a Hyperdrive config with caching.disabled: false, max_age: 5
2. Execute a complex query via query_typed (e.g., 5-CTE query with RANK() OVER, JOINs, CROSS JOIN)
3. Repeat 10 times — ~40-60% of requests fail with "unexpected message from server"
4. Set caching.disabled: true — 100% success rate immediately

[yonatan-genai]

Diagnostic update: failure happens in response stream, not initial handshake.

Instrumented tokio-postgres query_typed with per-message logging. The initial Parse→Bind→Describe→Execute→Sync handshake always succeeds (ParseComplete, BindComplete, ParameterDescription, RowDescription all arrive correctly). The "unexpected message from server" error occurs during RowStream polling — when reading DataRow messages from the response.

This is consistent with Hyperdrive releasing the backend connection mid-response-stream. The client sends a batched query, Hyperdrive forwards it to backend A, backend A starts sending DataRow messages, but Hyperdrive returns backend A to the pool before all DataRows are sent. Subsequent DataRow reads may hit a different backend or stale buffer.

Possibly related: prisma/prisma#23367 reports "Release called on client which has already been released to the pool" with Hyperdrive — same premature connection release pattern, different driver.

Wire-level evidence (from patched tokio-postgres on CF Workers):

• Successful request: msg[0] ParseComplete → msg[1] BindComplete → msg[2] ParameterDescription → msg[3] RowDescription → (RowStream reads DataRows) → CommandComplete → ReadyForQuery
• Failed request: msg[0]-[3] identical, then RowStream gets an unexpected message type instead of DataRow

[yonatan-genai]

Closing — will refile with cleaner reproduction steps and corrected root cause analysis.