[WIP] Cosmos: Cross-partition SELECT/WHERE queries with client-issued composite continuation tokens.#4391
Draft
analogrelay wants to merge 32 commits into
Conversation
Defines the design for feed operations (queries, read-many, change feed) in the driver crate, covering the Plan → Execute model, OperationPayload/OperationTarget refactors, continuation tokens, OpenTelemetry integration, and partition split handling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Incorporates Java SDK comparison findings: O(1) continuation tokens using nested ResumeState tree, unified execute_operation API, simplified OperationPlan enum (Trivial/MultiStep), restored PartitionKey targeting, ORDER BY filter injection via placeholder, and renamed execute_operation_pipeline to execute_single_operation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replaces §8 with Turn/Step/Request diagnostics structure. The driver provides structured timing and concurrency data via raw timestamps; the SDK computes derived values and creates OTEL spans. Introduces TurnDiagnostics and StepDiagnostics with enqueued/started/completed timestamps for concurrency observation. Computed fields (wait time, execution time, max concurrent steps) are omitted to minimize memory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address findings from 9 parallel review agents: - Fix stale execute_operation_pipeline reference in §5.3 diagram - Consolidate resume strategy: §5.5 now summarizes and points to §7.3 - Add backend query plan caching as §2 open issue - Note EffectivePartitionKey needs Ord for EpkRange<EPK> (§3.2) - Clarify phasing: spec complete at ReadMany; cross-partition query details are forward-looking and not locked - Use Arc<CosmosOperation> in PlanStep::Fetch for fan-out sharing - Add wall_clock_start/start_instant to TurnDiagnostics for OTEL - Add StepOutcome enum to StepDiagnostics for non-HTTP step errors - Minor grammar and code fence fixes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The cached operation plan already subsumes the backend query plan, which the Planner consumes during planning and does not need afterward. No separate caching mechanism is needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Introduce OperationTarget enum (None, PartitionKey, FeedRange) to replace the partition_key field on CosmosOperation. This models the three mutually-exclusive partition targeting modes at the type level: non-partitioned, single logical partition, and feed range. Create a driver-internal FeedRange type with EPK bounds for pipeline routing. Add SDK<->driver FeedRange conversion methods. Require OperationTarget in CosmosOperation::new() so all factory methods must explicitly specify their target. Update execute_operation to route based on target variant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`OperationOverrides`
Build the SequentialDrain pipeline node for cross-partition feed operations. It drains children left-to-right by EPK range, handling partition splits by splicing replacement nodes at the current position. - Add PageResult enum (Page/Drained/SplitRequired) and ChildNodes enum for correct VecDeque iteration - Add TopologyProvider trait and CachedTopologyProvider adapter backed by PartitionKeyRangeCache - Update Request node with EPK split recovery via TopologyProvider, returning SplitRequired to parent - Make TopologyProvider required on PipelineContext - Extract shared test mocks to dataflow/mocks.rs - 35 dataflow tests covering draining, splits, errors Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add backend query plan model types (QueryPlan, QueryInfo, QueryRange, etc.) for parsing gateway query plan responses. Implement build_pipeline in planner.rs that validates query plans (rejects unsupported features like top/limit/ordering/ aggregates/hybrid search) and constructs fan-out pipelines from backend-provided query ranges. Add plan_operation on CosmosDriver that handles trivial plans (non-query or single-partition) with singleton pipelines, and cross-partition queries by fetching a query plan from the backend then building a fan-out pipeline via build_pipeline. Supporting changes: - Add supported_query_features field to CosmosRequestHeaders - Add force_refresh parameter to CachedTopologyProvider - Add PartitionKeyRangeCache to CosmosDriver - Add fetch_partition_key_ranges and read_partition_key_ranges - Add OperationPlan public newtype wrapping Pipeline - Add Debug impl for Pipeline Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
analogrelay
force-pushed
the
ashleyst/select-and-where
branch
2 times, most recently
from
bfa69db to
4d44c7e
Compare
analogrelay
force-pushed
the
ashleyst/select-and-where
branch
from
4d44c7e to
60f89d3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.