feat(ilp): QWiP durable ack#14
Merged
Merged
Conversation
syncPing previously branched only on isDurableAck() and isSuccess(). Any error frame (parse, schema mismatch, security, internal, write error) arriving between PING and PONG was parsed into ackResponse, neither branch fired, and the error was silently discarded. A caller using ping() to confirm "all my batches landed" could get a false affirmative; the failure only surfaced on the next flush's waitForAck. Route error frames through inFlightWindow.fail so the next waitForAck / flush raises them, matching the normal waitForAck error-handling path. syncPing itself does not throw, so earlier durable/committed progress in the same ping round still reaches the caller. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…uestdb-client into ia_wal_upload_ack
Contributor
[PR Coverage check]😍 pass : 299 / 338 (88.46%) file detail
|
bluestreak01
approved these changes
Apr 25, 2026
bluestreak01
added a commit
that referenced
this pull request
Apr 27, 2026
Every frame written through the cursor SF path now carries its full schema definition and the complete symbol-dictionary delta starting at id 0. No schema-by-id refs, no incremental delta-dicts. The bytes persist on disk and get replayed against fresh server connections — post-reconnect, post-restart, or via background drainers adopting orphan slots. A frame whose schema ref points at an ID the new connection has never seen is unrecoverable; the spec's dedup-by-messageSequence assumption fixes duplicates, not stale refs. The previous testReplayResendsUnackedFramesAcrossReconnect only covered single-batch replay (first batch always carries full schemas) so the gap wasn't caught. Implementation: encode pass forces confirmedMaxId=-1 (full symbol delta from 0) and useSchemaRef=false (full schema definition, never a ref). Producer-side maxSentSchemaId / maxSentSymbolId tracking is now effectively dead state — left in place as a no-op safety net. Cost: bytes per batch grow vs the prior delta encoding. Acceptable for correctness across the entire recovery story. Test: SelfSufficientFramesTest sends two batches with distinct symbol values over the same connection and verifies batch 2 carries deltaStart=0 with deltaCount ≥ 2 — i.e. it redefines the prior batch's symbols too, instead of starting from where the prior delta left off. Spec updated with decision #14. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
STATUS_DURABLE_ACK(0x02) frame support to the QWP client. When a connection is opened withrequestDurableAck(true)(builder API orrequest_durable_ack=onconfig string), the client sends anX-QWP-Request-Durable-Ack: trueupgrade header. Servers with primary replication enabled then emit per-table durable-upload watermarks as WAL data reaches the object store.STATUS_OKandSTATUS_DURABLE_ACKframes now carry per-table entries (tableCount+[nameLen + name + seqTxn]repeating).STATUS_OKincludes a batch sequence number;STATUS_DURABLE_ACKdoes not (it is not tied to a specific client batch).QwpWebSocketSender:getHighestDurableSeqTxn(tableName)— highest seqTxn durably uploaded to object storegetHighestAckedSeqTxn(tableName)— highest seqTxn committed (written to WAL)ping()method that sends a WebSocket PING and drains pending frames (including durable ACKs and STATUS_OK) until the PONG arrives. The server flushes pending durable ACKs before replying, so afterping()returns the durable progress is up to date. Works in both sync and async (send-queue) modes.CharSequenceLongHashMapcollection for sync-mode per-table tracking.WebSocketResponse.durableAck()factory andisDurableAck()/getStatus()/getTableEntryCount()/getTableName(i)/getTableSeqTxn(i)accessors on the response model.Wire format changes
STATUS_OK (was:
status + sequence; now includes table entries):STATUS_DURABLE_ACK (new, no batch sequence):
Changed files
Sender—requestDurableAck(boolean)builder method,request_durable_ack=on|offconfig string parsingWebSocketClient—qwpRequestDurableAckflag →X-QWP-Request-Durable-Ack: trueheaderQwpWebSocketSender—setRequestDurableAck, per-tablegetHighestDurableSeqTxn/getHighestAckedSeqTxn,ping(), sync ping loop, durable-ack + table-entry handling inwaitForAckWebSocketResponse—STATUS_DURABLE_ACKconstant, per-table entry parsing/writing (readTableEntries/writeTableEntries/validateTableEntries), updated structural validationWebSocketSendQueue— per-tableConcurrentHashMap<String, SeqTxn>tracking for both committed and durable seqTxns,ping()/pingAndDrain/ PONG handling in I/O loopInFlightWindow—getHighestAckedSequence()accessorCharSequenceLongHashMap— new hash map for sync-mode per-table seqTxn trackingTest plan
WebSocketResponseTest— durable-ack factory, round-trip serialization, per-table entries on STATUS_OK and STATUS_DURABLE_ACK, structural validity, truncated entries, trailing garbage, empty table names, unknown status bytesQwpWebSocketSenderStateTest— default values for per-table durable/acked seqTxn,setRequestDurableAcklifecycle guards, sync ping processing (durable ACK, STATUS_OK, bare PONG),ping()after closeQwpWebSocketAckIntegrationTest— upgrade header sent/not-sent, sync durable ACK duringwaitForAck, STATUS_OK with table entries updates committed seqTxnsInFlightWindowTest—getHighestAckedSequenceinitial value, cumulative advancement, monotonicityWebSocketSendQueueTest— STATUS_OK with table entries updates committed seqTxn, durable-ack updates per-table seqTxn, monotonicity under out-of-order delivery, interleaving with STATUS_OK, ping blocks until PONG, ping with in-flight batches, ping timeout, ping transport error, initial value checkCharSequenceLongHashMapTest— put/get, update existing key, rehash, clear, contains, custom no-entry value, valueAt with keyIndex🤖 Generated with Claude Code