Skip to content

refactor: slim runtime and bind ORM through planner#349

Merged
KKould merged 21 commits into
mainfrom
refactor-runtime-orm-stack
Jun 14, 2026
Merged

refactor: slim runtime and bind ORM through planner#349
KKould merged 21 commits into
mainfrom
refactor-runtime-orm-stack

Conversation

@KKould

@KKould KKould commented Jun 12, 2026

Copy link
Copy Markdown
Member

Summary

This PR prepares KiteSQL 0.3.0 around a smaller embedded footprint and a cleaner planner/runtime boundary. The main change is not a single executor rewrite: it is a broad cleanup that feature-gates optional frontends, routes ORM queries through the normal binder/planner path, and centralizes execution metadata/codec state so less work is duplicated across storage backends and query shapes.

What Changed

  • Feature-gate optional components and dependencies: parser, shell, copy, time, decimal, python, rocksdb, and lmdb; remove the old pgwire server binary and net feature; make kitesql-shell an explicit shell feature target.
  • Rework database state around direct metadata caches plus a shared TableArena/PlanArena, so table, view, function, column, and index metadata can be referenced consistently during binding, optimization, and execution.
  • Refactor execution setup around ExecutionContext, ExecArena, local execution state, reusable TableCodec, projection scratch buffers, and DDL-apply records. Executor and transaction types are still generic where storage iterators require it, but most nodes now pull shared runtime state from the arena instead of carrying duplicate context.
  • Move SQL AST binding behind the parser feature and add binder-facing APIs that build LogicalPlan values directly.
  • Rewrite the ORM query/mutation path to use Database::bind / DBTransaction::bind with OrmContext, so typed ORM queries bind into core ScalarExpression and LogicalPlan structures instead of building SQL/parser-shaped intermediate paths.
  • Replace bincode/serde-based internal catalog and tuple serialization paths with project-local reference serialization and table-codec implementations, including buffer reuse for tuple decoding/encoding.
  • Clean up optimizer and expression execution around evaluator dispatch, column-pruning bookkeeping, parameterized index probes, subquery/set-membership handling, DESCRIBE column references, and allocation reduction.
  • Update README, ORM docs, examples, wasm tests, sqllogictest coverage, macro tests, and TPCC benchmark docs for the new APIs and feature layout.

Compatibility Notes

  • Bumps the crate version to 0.3.0.
  • The kite_sql pgwire server binary and net feature are removed in this PR.
  • parser remains enabled by default, but SQL parser support can now be left out of no-default-feature embedded builds.
  • orm now enables macros; parser, copy, decimal, time, python, shell, and storage backends are controlled by their own feature flags.
  • The documented ORM query-builder style changes from the previous from(...).eq(...).fetch(...) chain to binder-backed bind(|ctx| ...) plans.

Binary Size Comparison

Measured as stripped release binaries on x86_64-unknown-linux-gnu with rustc 1.88.0. The KiteSQL binary is a small ORM/typed API smoke example that creates a table, inserts two rows, fetches them, and drops the table.

Important: the PR measurements intentionally do not enable the parser feature. main still has parser/sqlparser as a non-optional dependency, so main cannot be measured with the same no-parser feature set.

Target Kind Features / scope Size
KiteSQL main c0e63a0 memory embedded database orm,macros; parser not feature-gated on main 7.25 MiB
KiteSQL PR 4ee5c66 memory embedded database orm; parser disabled 2.25 MiB
KiteSQL main c0e63a0 RocksDB embedded database orm,macros,rocksdb; parser not feature-gated on main 15.74 MiB
KiteSQL PR 4ee5c66 RocksDB embedded database orm,rocksdb; parser disabled 10.08 MiB
KiteSQL main c0e63a0 LMDB embedded database orm,macros,lmdb; parser not feature-gated on main 8.27 MiB
KiteSQL PR 4ee5c66 LMDB embedded database orm,lmdb; parser disabled 2.60 MiB
rusqlite 0.34.0 + bundled SQLite SQLite binding / embedded SQLite minimal in-memory SQL smoke app 1.31 MiB
SeaORM 2.0.0-rc.38 ORM framework, not a database minimal SQLite query-builder/model app, no DB engine linked 0.39 MiB
Diesel 2.3.10 + bundled SQLite ORM/query builder, not a database minimal SQLite ORM app with bundled libsqlite3 1.36 MiB
Turso embedded database 0.7.0-pre.7 and pinned 0.5.0 did not compile on this rustc due transitive/proc-macro API requirements n/a

KiteSQL PR reduction vs main in this measurement:

  • memory: 7.25 MiB -> 2.25 MiB, down 68.9%
  • RocksDB: 15.74 MiB -> 10.08 MiB, down 36.0%
  • LMDB: 8.27 MiB -> 2.60 MiB, down 68.5%

TPCC Throughput Comparison

Retested on the current local PR working tree after rebuilding target/release/tpcc without profiling features and clearing old TPCC database state for each backend. Each row is a fresh 1-warehouse run using TPCC's default 720-second measurement window. Raw logs were generated locally under tpcc/results/2026-06-14_720-full-current/ and are not committed. All runs passed TPCC constraint checks.

Backend Profile TpmC New-Order p90 Payment p90 Order-Status p90 Delivery p90 Stock-Level p90 Measure Time Status
KiteSQL LMDB embedded database / LMDB 61723 0.001s 0.001s 0.001s 0.002s 0.001s 720s ok
KiteSQL RocksDB embedded database / RocksDB 30446 0.001s 0.001s 0.001s 0.016s 0.002s 720s ok
SQLite balanced embedded SQLite / balanced profile 42989 0.001s 0.001s 0.001s 0.001s 0.001s 720s ok
SQLite practical embedded SQLite / practical profile 42276 0.001s 0.001s 0.001s 0.001s 0.001s 720s ok

These values are now mirrored in README.md and tpcc/README.md. The LMDB result is lower than the historical README value on the same machine, while RocksDB remains close to the historical value; the current run was kept as the source of truth because it is a full 720s matrix from clean databases.

Testing

  • cargo test --workspace
  • make wasm-build && make wasm-examples
  • Size smoke builds for memory/RocksDB/LMDB configurations with parser disabled
  • Full 720s TPCC matrix for KiteSQL LMDB, KiteSQL RocksDB, SQLite balanced, and SQLite practical

@KKould KKould self-assigned this Jun 12, 2026
@KKould KKould added the enhancement New feature or request label Jun 12, 2026
KKould added 2 commits June 13, 2026 03:50
Remove the transaction type parameter from executor nodes and route execution state through a runtime trait so executor code is compiled once instead of monomorphized per storage backend.
@KKould KKould changed the title Refactor runtime and ORM execution stack De-generic executor and refactor ORM stack Jun 12, 2026
@KKould KKould force-pushed the refactor-runtime-orm-stack branch from 44e2d3f to 2d22d96 Compare June 12, 2026 19:51
@KKould KKould changed the title De-generic executor and refactor ORM stack refactor: De-generic executor and refactor ORM stack Jun 13, 2026
@KKould KKould merged commit 96d8f74 into main Jun 14, 2026
14 checks passed
@KKould KKould changed the title refactor: De-generic executor and refactor ORM stack refactor: slim runtime and bind ORM through planner Jun 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant