Phase 8: query protocol — hardened wire decode, dual-surface AST, logical-plan IR, lowering, result/cursor encoding

Author: Thanga-Ganapathy

CHANGELOG.md

@@ -8,6 +8,48 @@ under a category (`Added` / `Changed` / `Fixed` / `Removed` / `Security`).
 
 ## [Unreleased]
 
+### Phase 8 — Query protocol, surfaces & IR
+
+#### Added
+- `proto`: the hardened wire layer — MessagePack bytes ⇄ a bounded `Doc`
+  tree under `DecodeLimits` (max message size checked up front, max depth
+  via explicit counter, max node count charged before any allocation);
+  rejects the reserved byte `0xC1`, ext types, non-string/duplicate map
+  keys, invalid UTF-8, over-`i64` ints, and trailing bytes (`DECISIONS.md`
+  D20).
+- `proto`: the typed query AST for **both surfaces** — pipeline stages
+  (scan/match/join/group/sort/project/distinct/limit/cursor) and the clause
+  form (from/joins/where/group_by/having/order_by/select/distinct/limit/
+  offset/cursor) — the full §5.2 expression grammar, and DML
+  (insert/update/delete/transaction/explain) with faithful selector
+  decoding (`where`/`{all:true}`/absent) for the Phase 9 validator.
+- `proto`: strict grammar enforcement at decode — unknown ops, stages,
+  expression nodes, and fields are typed `Validation` errors (queries are
+  data, never code); plus the canonical AST → wire encoding (decode ∘
+  encode = identity).
+- `proto`: protocol versioning — optional request `v` (missing = 1, other
+  values rejected); results always carry `v:1` (`DECISIONS.md` D21).
+- `proto`: the logical-plan IR (`Plan`): Scan, IndexScan (planner-only),
+  Filter, Join, Aggregate, Project, Distinct, Sort, Limit, Cursor.
+- `proto`: result encoding per `SPEC.md` §5.6 (`{v, ok, columns, rows,
+  cursor, applied, affected}`), the error-result shape (`{v, ok:false,
+  code, error}` with the §9 category as `code`), and the opaque cursor-token
+  envelope `[version][crc32c][payload]` with tamper rejection.
+- `query`: surface → IR lowering — the pipeline folds directly into a
+  `Plan`; the clause form desugars into its fixed-order pipeline and reuses
+  the same fold, making clause↔pipeline equivalence true by construction;
+  select-list aggregates become named `group` outputs (`DECISIONS.md` D22).
+- Exit-criteria tests: decode round-trips for every grammar node; oversized/
+  over-deep/over-budget messages rejected; a seeded 200k-input adversarial
+  fuzz suite over the decoder (random bytes + corpus mutations + hostile
+  container shapes, `DECISIONS.md` D23); clause↔pipeline equivalence on the
+  SPEC worked example plus 2 000 random generated pairs.
+
+#### Changed
+- `common`: gained the in-house CRC32C routine; `pager` now re-exports it
+  (`pager::crc32c` unchanged for callers) so `proto` can checksum cursor
+  tokens without depending on storage crates.
+
 ### Phase 7 — Indexing
 
 #### Added

DECISIONS.md

@@ -5,6 +5,92 @@ Per `PLAN.md` §1 rule 6, every resolution of an ambiguity or deviation from
 
 ---
 
+## D23 — Phase 8 fuzzing is a seeded in-house harness; libFuzzer waits for Phase 11
+
+**Phase:** 8 · **Status:** accepted
+
+`PLAN.md` Phase 8's exit demands "fuzzer clean on adversarial input", while
+the dedicated fuzz harness (cargo-fuzz/libFuzzer) is a Phase 11 deliverable.
+Pulling `cargo-fuzz` forward would add a nightly-only toolchain and external
+deps that D4's reasoning avoids.
+
+**Decision:** Phase 8 ships a deterministic, seeded adversarial-input suite
+(`proto/tests/proto_fuzz.rs`): 100k random byte strings, 100k corpus
+mutations (bit flips, truncations, overwrites, splices), and hand-built
+hostile container shapes (depth bombs, claimed-giant containers,
+str32/bin32 length lies). Every decoded request must also survive a
+canonical re-encode/re-decode round-trip. Coverage-guided fuzzing arrives
+with Phase 11's harness; this suite stays as the fast deterministic gate.
+
+## D22 — Clause desugaring semantics: aggregates, distinct, having
+
+**Phase:** 8 · **Status:** accepted
+
+`SPEC.md` §5.4 fixes the clause order (FROM → WHERE → GROUP → HAVING →
+PROJECT → ORDER → LIMIT) but leaves three details open. The clause lowerer
+desugars into a pipeline and reuses the pipeline fold, so equivalence is by
+construction; these rules define the desugaring:
+
+- **Aggregates in the clause `select` list** become named `group` outputs:
+  the alias names the output (`{as:["spent",{sum:…}]}` → agg `spent`), an
+  unaliased aggregate gets its function name (`{count:1}` → `count`), and a
+  name collision is a typed error. v1 allows aggregates only as the *whole*
+  select item (no `{add:[{sum:x},1]}`); grouping is implied by `group_by`
+  *or* select-list aggregates.
+- **`distinct` is an IR operator** (`Plan::Distinct`), placed after PROJECT
+  and before ORDER in the clause order. `ARCHITECTURE.md` §3.7's operator
+  list omits it though the stage exists in `SPEC.md` §5.3/§11; an explicit
+  operator beats desugaring into `Aggregate`, which would entangle lowering
+  with planning. (Addition, not contradiction — flagged per rule 6.)
+- **`having` without grouping** is rejected at lowering; `{distinct:false}`
+  is an explicit no-op stage. Structural shape (pipeline starts at a `scan`,
+  later sources arrive via `join`, `group` outputs really are aggregate
+  calls) is also enforced at lowering — names/types/§6 safety stay in the
+  Phase 9 validator.
+
+## D21 — Protocol version field and cursor-token envelope
+
+**Phase:** 8 · **Status:** accepted
+
+`PLAN.md` Phase 8 requires a protocol version field and a keyset cursor
+token; `SPEC.md` §5's grammar shows neither a version nor the token's bytes.
+
+**Decision:**
+- **Version:** requests may carry a top-level `v` (int). Missing means
+  version 1; any other value is a typed `UnsupportedVersion` error. Results
+  always carry `v:1`. Error results are `{v, ok:false, code, error}` with
+  `code` the stable `SPEC.md` §9 category identifier (the shape §5.6 leaves
+  implicit for the failure case).
+- **Cursor token:** opaque bytes `[version 0x01][crc32c(payload) BE][payload]`.
+  The payload (keyset position) is defined with the executor in Phase 9; the
+  envelope is fixed now so a truncated or mangled token is a clean
+  `Validation` error instead of a nonsense seek. CRC32C moved from `pager`
+  to `common` (same in-house routine, D3) so `proto` shares it without
+  depending on storage crates.
+
+## D20 — Two-stage hardened decode: bytes → bounded Doc tree → AST
+
+**Phase:** 8 · **Status:** accepted
+
+`ARCHITECTURE.md` §6 requires limits enforced *before* allocating and no
+unbounded recursion, but does not fix the decoder's architecture.
+
+**Decision:** decoding is two stages. A small hardened reader produces a
+generic `Doc` tree (null/bool/int/float/str/bin/array/map) under
+`DecodeLimits` — max message size (checked before reading), max depth
+(explicit counter), max node count (budget charged per node; container item
+counts are validated against both the remaining bytes and the remaining
+budget before any `Vec` allocation). It rejects the reserved byte `0xC1`,
+ext types, non-string and duplicate map keys, invalid UTF-8, ints outside
+`i64`, and trailing bytes. The AST mapping then works on the already-safe
+tree and enforces the grammar (unknown ops/stages/expressions/fields are
+typed errors). Defaults: 1 MiB / depth 64 (matching `types::MAX_JSON_DEPTH`)
+/ 100k nodes, embedder-configurable per `SPEC.md` §8. The node cap bounds
+the intermediate tree's memory, so the two-stage shape costs nothing
+adversarially and keeps all byte-level hardening in one ~200-line module.
+Insert-row values that are containers become `json` values (re-encoded
+canonical MessagePack), matching §5.5's `data:{role:"admin"}` example.
+
 ## D19 — Reclamation only runs inside a committing batch
 
 **Phase:** 7 · **Status:** accepted (bug fix of Phase 4 behavior)

crates/pager/src/crc.rs crates/common/src/crc.rscrates/pager/src/crc.rs renamed to crates/common/src/crc.rs

@@ -2,7 +2,8 @@
 //!
 //! Implemented in-house rather than pulled as a dependency: it is a small,
 //! well-understood, non-security-sensitive integrity check (used to detect page
-//! corruption, not to resist tampering). See `DECISIONS.md` D3.
+//! corruption and mangled cursor tokens, not to resist tampering). See
+//! `DECISIONS.md` D3. Lives in `common` so both `pager` and `proto` share it.
 
 /// Reflected Castagnoli polynomial (0x1EDC6F41 reflected).
 const POLY: u32 = 0x82F6_3B78;
@@ -35,7 +36,7 @@ static TABLE: [u32; 256] = build_table();
 ///
 /// ```
 /// // Standard CRC32C check value for the ASCII string "123456789".
-/// assert_eq!(pager::crc32c(b"123456789"), 0xE306_9283);
+/// assert_eq!(common::crc32c(b"123456789"), 0xE306_9283);
 /// ```
 pub fn crc32c(data: &[u8]) -> u32 {
     let mut crc = 0xFFFF_FFFFu32;

crates/common/src/lib.rs

@@ -4,20 +4,24 @@
 //! (see `ARCHITECTURE.md` §2). It holds only genuinely shared abstractions:
 //!
 //! - the [`ErrorCategory`] taxonomy from `SPEC.md` §9 and the
-//!   [`CategorizedError`] trait each crate's error enum implements, and
+//!   [`CategorizedError`] trait each crate's error enum implements,
 //! - the injectable host services [`Clock`], [`Rng`], and [`IoBackend`] (with
 //!   real-file, in-memory, and fault-injecting backends) that make the lower
-//!   layers testable and deterministically simulatable from day one.
+//!   layers testable and deterministically simulatable from day one, and
+//! - the in-house [`crc32c`] checksum shared by `pager` (page integrity) and
+//!   `proto` (cursor-token integrity).
 //!
 //! Domain newtypes (`PageId`, `TxnId`, `Value`, …) intentionally live in their
 //! owning crates, not here — `common` is not a junk drawer.
 
 mod clock;
+mod crc;
 mod error;
 mod io;
 mod rng;
 
 pub use clock::{Clock, ManualClock, SystemClock};
+pub use crc::crc32c;
 pub use error::{CategorizedError, ErrorCategory};
 pub use io::{FaultInjectingBackend, FaultPoint, IoBackend, IoError, IoResult, MemoryBackend};
 pub use rng::{Rng, SeededRng};

crates/dbms/src/lib.rs

@@ -93,7 +93,7 @@ mod tests {
 
     #[test]
     fn protocol_malformed_is_validation() {
-        let err: Error = proto::ProtoError::Malformed.into();
+        let err: Error = proto::ProtoError::Truncated.into();
         assert_eq!(err.category(), ErrorCategory::Validation);
     }
 

crates/pager/src/lib.rs

@@ -19,15 +19,14 @@
 //! ```
 
 mod cache;
-mod crc;
 mod freelist;
 mod meta;
 mod page;
 mod pager;
 
 use common::{CategorizedError, ErrorCategory};
 
-pub use crc::crc32c;
+pub use common::crc32c;
 pub use meta::Meta;
 pub use page::{Frame, PageType, HEADER_SIZE, PAGE_PAYLOAD_SIZE, PAGE_SIZE};
 pub use pager::{Pager, PagerStats, DEFAULT_CACHE_BYTES};

crates/pager/src/page.rs

@@ -105,7 +105,7 @@ pub fn page_id(frame: &Frame) -> u64 {
 pub fn verify(bytes: &[u8], expected: PageId) -> Result<&Frame> {
     let frame = as_frame(bytes, expected)?;
     let stored = read_u32(frame, CRC_OFF);
-    let computed = crate::crc::crc32c(&frame[TYPE_OFF..]);
+    let computed = common::crc32c(&frame[TYPE_OFF..]);
     if stored != computed {
         return Err(corrupt(CorruptionKind::Checksum {
             page: expected.get(),
@@ -129,7 +129,7 @@ pub fn finalize(frame: &mut Frame, page_type: PageType, id: PageId) {
     frame[6] = 0;
     frame[7] = 0;
     write_u64(frame, ID_OFF, id.get());
-    let crc = crate::crc::crc32c(&frame[TYPE_OFF..]);
+    let crc = common::crc32c(&frame[TYPE_OFF..]);
     write_u32(frame, CRC_OFF, crc);
 }
 
@@ -203,7 +203,7 @@ mod tests {
         finalize(&mut frame, PageType::Data, PageId::new(2));
         frame[TYPE_OFF] = 99;
         // Re-checksum so the type byte is what fails, not the CRC.
-        let crc = crate::crc::crc32c(&frame[TYPE_OFF..]);
+        let crc = common::crc32c(&frame[TYPE_OFF..]);
         write_u32(&mut frame, CRC_OFF, crc);
         let verified = verify(&frame[..], PageId::new(2)).unwrap();
         assert!(matches!(