feat(cubeorchestrator): Use Arrow format for CubeStore response format#10949
feat(cubeorchestrator): Use Arrow format for CubeStore response format#10949ovr wants to merge 2 commits into
Conversation
|
Claude finished @ovr's task in 4m 41s —— View job Code Review: Arrow Format for CubeStore Response
SummaryWell-structured PR that adds Arrow IPC stream decoding into the existing columnar Issues Found1 High-severity bug, 2 medium items, rest are low/informational.
What's done well
Testing gaps
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## feat/release-v1.7.0 #10949 +/- ##
====================================================
Coverage 79.30% 79.30%
====================================================
Files 472 472
Lines 96880 96880
Branches 3539 3539
====================================================
Hits 76826 76826
Misses 19548 19548
Partials 506 506
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
17ec119 to
38ab4f2
Compare
|
Claude finished @ovr's task in 4m 13s —— View job Code Review: Arrow Format for CubeStore Response (Re-review after 6 commits)
SummaryGood iteration on the previous review. The follow-up commits addressed the key robustness concerns: downcast macros now use Two new issues found in this pass: Issues Found
What improved since last review
Testing observation
|
|
Claude finished @ovr's task in 2m 53s —— View job Code Review: Arrow Format for CubeStore Response (Re-review after 8 commits)
SummaryGood iteration. The latest commit ( Remaining Issue
This was flagged in the previous review and is still present. I've posted an inline comment with a suggested fix. What was resolved since last review
Overall AssessmentThe PR is in good shape. The Arrow IPC decoding path is well-implemented with comprehensive type coverage, safe downcast error handling, and proper backward compatibility. The single remaining issue ( |
7eab593 to
f46307d
Compare
|
Claude finished @ovr's task in 2m 31s —— View job Code Review: Arrow Format for CubeStore Response (Re-review after sync)
SummaryThe branch was rebased (6 commits, down from 8 in the prior review). The good improvements from earlier rounds are preserved — safe downcast handling ( Re-affirmed 2 prior unresolved threads that still apply (both on Issues Found
Both have inline comments with suggested fixes. What's in good shape
Testing observation
|
|
Claude finished @ovr's task in 1m 1s —— View job No new issues found. PR remains in good shape.Re-review after sync — 2 new commits since last review (
|
|
Claude finished @ovr's task in 33s —— View job No new issues found. PR remains in good shape.Re-review after sync — new commit
|
|
Claude finished @ovr's task in 35s —— View job No new issues found. PR remains in good shape.Re-review after sync — new commit
|
|
I'll analyze this and get back to you. |
fix(cubeorchestrator): parse time strings with variable-length fraction `transform_value` normalized time-typed strings using the fixed-width `%.3f` parse format. Since chrono 0.4.42 (PR #1705) `%.3f` requires exactly three fractional digits on parse, so a timestamp string carrying a different count — e.g. the JDBC bridge's `java.sql.Timestamp::toString()` rendering `2020-04-01 00:00:00.0` — fails to parse and falls through unnormalized, leaking `2020-04-01 00:00:00.0` instead of `2020-04-01T00:00:00.000`. The chrono bump rode in with the new Arrow crates on this branch; the parse block itself was unchanged. Switch the parse formats to `%.f` (variable-length fraction), which accepts any digit count (including none) in every chrono version. The output is still reformatted to `%.3f`, so normalized values are unchanged. feat(cubeorchestrator): support Decimal32 and Decimal64 arrow arrays chore: try to fix chore: decimal formatting chore: tmp chore: fix
|
Claude finished @ovr's task in 56s —— View job No new issues found. PR remains in good shape.Re-review after squash (2 commits:
|
| Area | Location | Verdict |
|---|---|---|
?? default for responseFormat |
CubeStoreDriver.ts:120 |
Nullish coalescing preserved; capability-gated (Arrow only for CubeStore ≥ 1.6.66) |
| Downcast safety | query_message_parser.rs:347 |
ok_or_else → ParseError::ArrowError (no .unwrap()) |
from_arrow visibility |
query_message_parser.rs:279 |
pub(crate) — correct; benchmarks use public from_cubestore_fb |
| Numeric fidelity + wire shape | query_result_transform.rs:1196-1201 |
Int64/UInt64/Float64/Timestamp all render via collect_str — legacy string wire shape preserved |
| Timestamp formatting | query_result_transform.rs:1169,1201,1317 |
TIMESTAMP_ITEMS const shared by Display + Serialize |
| Size guard | query_result_transform.rs:1391 |
32-byte enum layout still enforced |
| Completed payload | query_message_parser.rs:218 |
HttpQueryResultArrow handled; HttpQueryResultCompleted → QueryResult::empty() |
| Decimal handling | query_message_parser.rs |
Custom decimal_to_string for i32/i64/i128/i256; Decimal32/64 rejected due to CubeStore bitWidth compat hack |
| Test coverage | query_message_parser.rs:690-855 |
Basic types, Decimal256 beyond i128, leading-null decimal, unsupported types, Completed round-trip |
Testing observation (unchanged)
transform_value reformats only DBResponsePrimitive::String when type_ == "time"; Timestamp(NaiveDateTime) passes through via other => other and relies on the dedicated Serialize impl. Correct but not directly tested — a single assertion would document the intentional bypass. Non-blocking.
Release-notes reminder
v1.7.0 renders numeric measure/dimension values as JSON strings rather than JSON numbers. Callers doing typeof val === 'number' or arithmetic without coercion will break. Decimal columns strip trailing fractional zeros (e.g. 1.5, not 1.500). Worth a prominent changelog call-out.
• Branch •
|
Claude finished @ovr's task in 33s —— View job No new issues found. PR remains in good shape.Re-review after sync — 2 commits atop base (no substantive changes since the prior squash review)
State of the branchThe branch is squashed to 2 commits atop the base:
All 15+ commits of iteration are folded into these two. No production-code deltas relative to the last review pass. Verified end-to-end
Prior threadsAll 19 prior review threads from earlier rounds remain correctly resolved against the current diff. Testing observation (unchanged)
Release-notes reminderv1.7.0 renders numeric measure/dimension values (both) as JSON strings rather than JSON numbers. Callers doing • Branch • |
|
Claude finished @ovr's task in 39s —— View job No new issues found. PR remains in good shape.Re-review after sync — no substantive changes since last pass
State of the branch2 commits atop the base, same content as the prior review:
No code deltas since the last review pass. Verified end-to-end
Prior threadsAll 19 prior review threads from earlier rounds remain correctly resolved. Release-notes reminderv1.7.0 renders numeric measure/dimension values as JSON strings rather than JSON numbers. Callers doing • Branch • |
…trings, fix #1879 For a long time, Cube returned inconsistent JSON types for numeric values depending on the driver — sometimes strings, sometimes numbers. This is incorrect: the wire type of a value should not depend on the data source. We have decided to return all numeric values as strings. The legacy CubeStore result set carried every value as a string, so numbers always serialized to JSON strings. The Arrow path produces real Int64/UInt64/Float64 primitives that serialized to JSON numbers, which changed the wire shape and broke pre-aggregation snapshots (e.g. "5" -> 5, "0.6666666666666666" -> 0.6666666666666666). BREAKING CHANGE: All numeric values in query results are now serialized as JSON strings regardless of the data source driver. Previously the JSON type of numeric values varied by driver (string or number). Clients that relied on receiving JSON numbers must parse the string values instead.
|
Claude finished @ovr's task in 39s —— View job No new issues found. PR remains in good shape.Re-review after sync — no substantive changes since last pass
State of the branch2 commits atop the base:
No production-code deltas since the last review pass — the tip SHA changed from Verified end-to-end
Prior threadsAll 19 prior review threads from earlier rounds remain correctly resolved. Release-notes reminderv1.7.0 renders numeric measure/dimension values as JSON strings rather than JSON numbers. Callers doing • Branch • |

Add QueryResult::from_arrow to decode Arrow IPC stream payloads into the columnar QueryResult, and route the HttpQueryResult (Arrow) command in from_cubestore_fb to it. Arrow temporal columns map to a new dedicated DBResponsePrimitive::Timestamp variant that serializes to the existing ISO-8601 millisecond format.
BREAKING CHANGE: all numeric values in query results are now serialized as JSON strings regardless of the data source driver. For a long time, Cube returned inconsistent JSON types for numeric values depending on the driver — sometimes strings, sometimes numbers. The wire type of a value should not depend on the data source, so we standardize on strings. Clients that relied on receiving JSON numbers must parse the string values instead (front-end SDKs already provide the
castNumericsoption for this).Fix #1879