feat: Upgrade DataFusion to 50.3.0 (#66)

* Adding dev containers

* Bumping to DataFusion 50.3.0

* fix: Add missing low_memory parameter to reset_state()

After merging master (aa10539) which introduced the low_memory feature,
the reset_state() method needed to be updated to pass self.low_memory
when calling try_new().

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Archiving openspec spec

* Change to self-hosted runners

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: mwiewior

.claude/commands/openspec/apply.md

@@ -0,0 +1,23 @@
+---
+name: OpenSpec: Apply
+description: Implement an approved OpenSpec change and keep tasks in sync.
+category: OpenSpec
+tags: [openspec, apply]
+---
+<!-- OPENSPEC:START -->
+**Guardrails**
+- Favor straightforward, minimal implementations first and add complexity only when it is requested or clearly required.
+- Keep changes tightly scoped to the requested outcome.
+- Refer to `openspec/AGENTS.md` (located inside the `openspec/` directory—run `ls openspec` or `openspec update` if you don't see it) if you need additional OpenSpec conventions or clarifications.
+
+**Steps**
+Track these steps as TODOs and complete them one by one.
+1. Read `changes/<id>/proposal.md`, `design.md` (if present), and `tasks.md` to confirm scope and acceptance criteria.
+2. Work through tasks sequentially, keeping edits minimal and focused on the requested change.
+3. Confirm completion before updating statuses—make sure every item in `tasks.md` is finished.
+4. Update the checklist after all work is done so each task is marked `- [x]` and reflects reality.
+5. Reference `openspec list` or `openspec show <item>` when additional context is required.
+
+**Reference**
+- Use `openspec show <id> --json --deltas-only` if you need additional context from the proposal while implementing.
+<!-- OPENSPEC:END -->

.claude/commands/openspec/archive.md

@@ -0,0 +1,27 @@
+---
+name: OpenSpec: Archive
+description: Archive a deployed OpenSpec change and update specs.
+category: OpenSpec
+tags: [openspec, archive]
+---
+<!-- OPENSPEC:START -->
+**Guardrails**
+- Favor straightforward, minimal implementations first and add complexity only when it is requested or clearly required.
+- Keep changes tightly scoped to the requested outcome.
+- Refer to `openspec/AGENTS.md` (located inside the `openspec/` directory—run `ls openspec` or `openspec update` if you don't see it) if you need additional OpenSpec conventions or clarifications.
+
+**Steps**
+1. Determine the change ID to archive:
+   - If this prompt already includes a specific change ID (for example inside a `<ChangeId>` block populated by slash-command arguments), use that value after trimming whitespace.
+   - If the conversation references a change loosely (for example by title or summary), run `openspec list` to surface likely IDs, share the relevant candidates, and confirm which one the user intends.
+   - Otherwise, review the conversation, run `openspec list`, and ask the user which change to archive; wait for a confirmed change ID before proceeding.
+   - If you still cannot identify a single change ID, stop and tell the user you cannot archive anything yet.
+2. Validate the change ID by running `openspec list` (or `openspec show <id>`) and stop if the change is missing, already archived, or otherwise not ready to archive.
+3. Run `openspec archive <id> --yes` so the CLI moves the change and applies spec updates without prompts (use `--skip-specs` only for tooling-only work).
+4. Review the command output to confirm the target specs were updated and the change landed in `changes/archive/`.
+5. Validate with `openspec validate --strict` and inspect with `openspec show <id>` if anything looks off.
+
+**Reference**
+- Use `openspec list` to confirm change IDs before archiving.
+- Inspect refreshed specs with `openspec list --specs` and address any validation issues before handing off.
+<!-- OPENSPEC:END -->

.claude/commands/openspec/proposal.md

@@ -0,0 +1,27 @@
+---
+name: OpenSpec: Proposal
+description: Scaffold a new OpenSpec change and validate strictly.
+category: OpenSpec
+tags: [openspec, change]
+---
+<!-- OPENSPEC:START -->
+**Guardrails**
+- Favor straightforward, minimal implementations first and add complexity only when it is requested or clearly required.
+- Keep changes tightly scoped to the requested outcome.
+- Refer to `openspec/AGENTS.md` (located inside the `openspec/` directory—run `ls openspec` or `openspec update` if you don't see it) if you need additional OpenSpec conventions or clarifications.
+- Identify any vague or ambiguous details and ask the necessary follow-up questions before editing files.
+
+**Steps**
+1. Review `openspec/project.md`, run `openspec list` and `openspec list --specs`, and inspect related code or docs (e.g., via `rg`/`ls`) to ground the proposal in current behaviour; note any gaps that require clarification.
+2. Choose a unique verb-led `change-id` and scaffold `proposal.md`, `tasks.md`, and `design.md` (when needed) under `openspec/changes/<id>/`.
+3. Map the change into concrete capabilities or requirements, breaking multi-scope efforts into distinct spec deltas with clear relationships and sequencing.
+4. Capture architectural reasoning in `design.md` when the solution spans multiple systems, introduces new patterns, or demands trade-off discussion before committing to specs.
+5. Draft spec deltas in `changes/<id>/specs/<capability>/spec.md` (one folder per capability) using `## ADDED|MODIFIED|REMOVED Requirements` with at least one `#### Scenario:` per requirement and cross-reference related capabilities when relevant.
+6. Draft `tasks.md` as an ordered list of small, verifiable work items that deliver user-visible progress, include validation (tests, tooling), and highlight dependencies or parallelizable work.
+7. Validate with `openspec validate <id> --strict` and resolve every issue before sharing the proposal.
+
+**Reference**
+- Use `openspec show <id> --json --deltas-only` or `openspec show <spec> --type spec` to inspect details when validation fails.
+- Search existing requirements with `rg -n "Requirement:|Scenario:" openspec/specs` before writing new ones.
+- Explore the codebase with `rg <keyword>`, `ls`, or direct file reads so proposals align with current implementation realities.
+<!-- OPENSPEC:END -->

.claude/settings.local.json

@@ -0,0 +1,28 @@
+{
+  "permissions": {
+    "allow": [
+      "WebFetch(domain:github.com)",
+      "WebFetch(domain:ml-explore.github.io)",
+      "Bash(cargo check:*)",
+      "WebFetch(domain:docs.rs)",
+      "Bash(cargo doc:*)",
+      "Bash(cargo test:*)",
+      "Bash(cargo bench:*)",
+      "Bash(timeout 120s cargo bench --bench quick_optimization_test)",
+      "Bash(git add src/physical_planner/joins/interval_join.rs Cargo.toml benches/datafusion_optimized_benchmark.rs)",
+      "Bash(grep:*)",
+      "Bash(sed:*)",
+      "Bash(git commit:*)",
+      "Bash(git cherry-pick:*)",
+      "Bash(git add:*)",
+      "Bash(git reset:*)",
+      "Bash(git checkout:*)",
+      "Bash(git unstage:*)",
+      "Bash(pre-commit run:*)",
+      "Bash(python3:*)",
+      "Bash(git log:*)"
+    ],
+    "deny": [],
+    "ask": []
+  }
+}

.devcontainer/Dockerfile

@@ -0,0 +1,20 @@
+FROM mcr.microsoft.com/devcontainers/rust:2-1-bullseye
+
+ARG CLAUDE_CODE_VERSION=latest
+
+USER root
+
+# Install Node.js 24.x (latest LTS) from NodeSource
+RUN apt-get update && \
+    apt-get install -y ca-certificates curl gnupg && \
+    mkdir -p /etc/apt/keyrings && \
+    curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg && \
+    echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_24.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list && \
+    apt-get update && \
+    apt-get install -y nodejs && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+RUN npm install -g @anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}
+RUN npm install -g @fission-ai/openspec@latest
+USER vscode

.devcontainer/devcontainer.json

@@ -0,0 +1,44 @@
+// For format details, see https://aka.ms/devcontainer.json. For config options, see the
+// README at: https://github.com/devcontainers/templates/tree/main/src/rust
+{
+	"name": "Rust",
+	"dockerFile": "Dockerfile",
+  // Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
+//	"image": "mcr.microsoft.com/devcontainers/rust:2-1-bullseye",
+
+	// Use 'mounts' to make the cargo cache persistent in a Docker Volume.
+	// "mounts": [
+	// 	{
+	// 		"source": "devcontainer-cargo-cache-${devcontainerId}",
+	// 		"target": "/usr/local/cargo",
+	// 		"type": "volume"
+	// 	}
+	// ]
+
+	// Features to add to the dev container. More info: https://containers.dev/features.
+	// "features": {},
+
+	// Use 'forwardPorts' to make a list of ports inside the container available locally.
+	// "forwardPorts": [],
+
+	// Use 'postCreateCommand' to run commands after the container is created.
+	// "postCreateCommand": "rustc --version",
+
+	// Configure tool-specific properties.
+  "customizations" : {
+    "jetbrains" : {
+      "settings": {
+        "com.intellij.database:app:DatabaseSettings.enable-local-filter-by-default": false,
+        "com.intellij:app:BaseRefactoringSettings.safe_delete_when_delete": false,
+        "com.intellij:app:BaseRefactoringSettings.rename_search_in_comments_for_file": false,
+        "com.intellij:app:BaseRefactoringSettings.rename_search_for_references_for_file": false,
+        "com.intellij:app:BaseRefactoringSettings.rename_search_for_references_for_directory": false,
+        "com.intellij:app:BaseRefactoringSettings.move_search_for_references_for_file": false
+      },
+      "backend" : "RustRover"
+    }
+  },
+
+	// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
+	// "remoteUser": "root"
+}

.github/workflows/ci.yml

@@ -16,7 +16,7 @@ env:
 jobs:
   build-and-test:
     name: build and test
-    runs-on: [self-hosted, Linux, openstack-ii]
+    runs-on: [ubuntu-latest]
     steps:
       - uses: actions/checkout@v4
       - name: Install Rust
@@ -37,7 +37,7 @@ jobs:
 
   fmt:
     name: fmt
-    runs-on: [self-hosted, Linux, openstack-ii]
+    runs-on: [ubuntu-latest]
     steps:
       - uses: actions/checkout@v4
       - name: Install Rust

.pre-commit-config.yaml

@@ -1,19 +1,20 @@
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.4.0
+    rev: v6.0.0
     hooks:
       - id: check-yaml
-        stages: [commit]
+        stages: [pre-commit]
       - id: check-json
-        stages: [commit]
+        stages: [pre-commit]
+        exclude: '^\.devcontainer/devcontainer\.json$'
       - id: check-toml
-        stages: [commit]
+        stages: [pre-commit]
       - id: check-merge-conflict
-        stages: [commit]
+        stages: [pre-commit]
       - id: check-case-conflict
-        stages: [commit]
+        stages: [pre-commit]
       - id: detect-private-key
-        stages: [commit]
+        stages: [pre-commit]
 
   - repo: https://github.com/doublify/pre-commit-rust
     rev: v1.0

AGENTS.md

@@ -0,0 +1,60 @@
+<!-- OPENSPEC:START -->
+# OpenSpec Instructions
+
+These instructions are for AI assistants working in this project.
+
+Always open `@/openspec/AGENTS.md` when the request:
+- Mentions planning or proposals (words like proposal, spec, change, plan)
+- Introduces new capabilities, breaking changes, architecture shifts, or big performance/security work
+- Sounds ambiguous and you need the authoritative spec before coding
+
+Use `@/openspec/AGENTS.md` to learn:
+- How to create and apply change proposals
+- Spec format and conventions
+- Project structure and guidelines
+
+Keep this managed block so 'openspec update' can refresh the instructions.
+
+<!-- OPENSPEC:END -->
+
+# Repository Guidelines
+
+## Project Structure & Module Organization
+- Workspace: Rust 2021, crates in `sequila/`.
+- Core library: `sequila/sequila-core` — DataFusion extensions, physical planner, and interval-join optimization.
+- CLI: `sequila/sequila-cli` — REPL and file execution for SQL.
+- Tests: unit/integration under `sequila/sequila-core/tests` and crate-local tests; benches in `sequila/sequila-core/benches`.
+- Utilities/data: `queries/` for sample SQL, `testing/data/` for local datasets, `bin/env.sh` for env helpers.
+
+## Build, Test, and Development Commands
+- Build: `cargo build` (release: `cargo build --release`).
+- Run CLI: `RUST_LOG=info cargo run -p sequila-cli -- --file queries/q1-coitrees.sql`.
+- REPL: `RUST_LOG=info cargo run -p sequila-cli`.
+- Tests: `cargo test --workspace` (async tests use Tokio).
+- Benchmarks: `RUSTFLAGS="-Ctarget-cpu=native" cargo bench --bench databio_benchmark -- --quick`.
+
+## Coding Style & Naming Conventions
+- Formatter: rustfmt via pre-commit. Run `cargo fmt --all` before pushing.
+- Lints: `cargo check` runs in pre-commit; keep builds warning-free.
+- Naming: crates/modules `snake_case`, types/enums `PascalCase`, functions/vars `snake_case`, constants `SCREAMING_SNAKE_CASE`.
+- Indentation: 4 spaces; avoid long lines unless readability benefits.
+
+## Testing Guidelines
+- Frameworks: Rust built-in tests, `rstest` for parametrization, `tokio::test` for async.
+- Locations: unit tests inline in `src/`; integration tests in `sequila-core/tests/`.
+- Conventions: name tests descriptively (e.g., `interval_rule_eq`); include edge cases for join predicates and config options.
+- Coverage: no strict target; add tests for new features and bug fixes.
+
+## Commit & Pull Request Guidelines
+- Commits: imperative mood, concise title; optionally use prefixes (`feat:`, `fix:`, `perf:`). Reference issues/PRs where relevant.
+- PRs: include scope/intent, key changes, how to test (commands/queries), and performance notes if applicable. Add screenshots/log snippets when helpful.
+- Keep diffs focused; update README or examples when behavior changes.
+
+## Architecture & Configuration Tips
+- Core idea: replace/augment DataFusion planning with `SeQuiLaQueryPlanner` and an interval-join physical optimization rule.
+- Useful SQL settings:
+  - `SET sequila.prefer_interval_join TO true;`
+  - `SET sequila.interval_join_algorithm TO coitrees;`
+  - `SET datafusion.optimizer.repartition_joins TO false;`
+- Bench data: export `BENCH_DATA_ROOT` to point at local datasets.
+

CLAUDE.md

@@ -0,0 +1,18 @@
+<!-- OPENSPEC:START -->
+# OpenSpec Instructions
+
+These instructions are for AI assistants working in this project.
+
+Always open `@/openspec/AGENTS.md` when the request:
+- Mentions planning or proposals (words like proposal, spec, change, plan)
+- Introduces new capabilities, breaking changes, architecture shifts, or big performance/security work
+- Sounds ambiguous and you need the authoritative spec before coding
+
+Use `@/openspec/AGENTS.md` to learn:
+- How to create and apply change proposals
+- Spec format and conventions
+- Project structure and guidelines
+
+Keep this managed block so 'openspec update' can refresh the instructions.
+
+<!-- OPENSPEC:END -->